[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260503 13:57:05.187278 8299 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.8.26.254:44623
I20260503 13:57:05.188788 8299 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260503 13:57:05.189797 8299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260503 13:57:05.203753 8305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:57:05.203855 8308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:57:05.203868 8306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:57:06.300180 8307 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260503 13:57:06.300280 8299 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260503 13:57:06.305173 8299 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260503 13:57:06.305405 8299 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260503 13:57:06.305594 8299 hybrid_clock.cc:648] HybridClock initialized: now 1777816626305571 us; error 0 us; skew 500 ppm
I20260503 13:57:06.306404 8299 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260503 13:57:06.311332 8299 webserver.cc:492] Webserver started at http://127.8.26.254:35047/ using document root <none> and password file <none>
I20260503 13:57:06.312165 8299 fs_manager.cc:362] Metadata directory not provided
I20260503 13:57:06.312402 8299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260503 13:57:06.312860 8299 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260503 13:57:06.318624 8299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/master-0-root/instance:
uuid: "b5336cc542d14c8ab2cc89f8c8b9dde2"
format_stamp: "Formatted at 2026-05-03 13:57:06 on dist-test-slave-jcqq"
I20260503 13:57:06.326745 8299 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260503 13:57:06.331893 8314 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:06.333019 8299 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260503 13:57:06.333360 8299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/master-0-root
uuid: "b5336cc542d14c8ab2cc89f8c8b9dde2"
format_stamp: "Formatted at 2026-05-03 13:57:06 on dist-test-slave-jcqq"
I20260503 13:57:06.333676 8299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260503 13:57:06.398765 8299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260503 13:57:06.400242 8299 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260503 13:57:06.400676 8299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260503 13:57:06.468669 8299 rpc_server.cc:307] RPC server started. Bound to: 127.8.26.254:44623
I20260503 13:57:06.468740 8365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.26.254:44623 every 8 connection(s)
I20260503 13:57:06.476341 8366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:06.496760 8366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2: Bootstrap starting.
I20260503 13:57:06.502183 8366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:06.503871 8366 log.cc:826] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2: Log is configured to *not* fsync() on all Append() calls
I20260503 13:57:06.507958 8366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2: No bootstrap required, opened a new log
I20260503 13:57:06.527837 8366 raft_consensus.cc:359] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5336cc542d14c8ab2cc89f8c8b9dde2" member_type: VOTER }
I20260503 13:57:06.528362 8366 raft_consensus.cc:385] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:06.528621 8366 raft_consensus.cc:740] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5336cc542d14c8ab2cc89f8c8b9dde2, State: Initialized, Role: FOLLOWER
I20260503 13:57:06.529439 8366 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5336cc542d14c8ab2cc89f8c8b9dde2" member_type: VOTER }
I20260503 13:57:06.529933 8366 raft_consensus.cc:399] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260503 13:57:06.530206 8366 raft_consensus.cc:493] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260503 13:57:06.530521 8366 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:06.628418 8366 raft_consensus.cc:515] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5336cc542d14c8ab2cc89f8c8b9dde2" member_type: VOTER }
I20260503 13:57:06.629307 8366 leader_election.cc:304] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5336cc542d14c8ab2cc89f8c8b9dde2; no voters:
I20260503 13:57:06.631096 8366 leader_election.cc:290] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260503 13:57:06.631378 8369 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:06.633742 8369 raft_consensus.cc:697] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [term 1 LEADER]: Becoming Leader. State: Replica: b5336cc542d14c8ab2cc89f8c8b9dde2, State: Running, Role: LEADER
I20260503 13:57:06.634877 8369 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5336cc542d14c8ab2cc89f8c8b9dde2" member_type: VOTER }
I20260503 13:57:06.636021 8366 sys_catalog.cc:565] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [sys.catalog]: configured and running, proceeding with master startup.
I20260503 13:57:06.644600 8370 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5336cc542d14c8ab2cc89f8c8b9dde2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5336cc542d14c8ab2cc89f8c8b9dde2" member_type: VOTER } }
I20260503 13:57:06.644829 8371 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5336cc542d14c8ab2cc89f8c8b9dde2. Latest consensus state: current_term: 1 leader_uuid: "b5336cc542d14c8ab2cc89f8c8b9dde2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5336cc542d14c8ab2cc89f8c8b9dde2" member_type: VOTER } }
I20260503 13:57:06.645431 8370 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [sys.catalog]: This master's current role is: LEADER
I20260503 13:57:06.645424 8371 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [sys.catalog]: This master's current role is: LEADER
I20260503 13:57:06.649453 8378 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260503 13:57:06.660159 8378 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260503 13:57:06.662133 8299 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260503 13:57:06.677209 8378 catalog_manager.cc:1357] Generated new cluster ID: 330b61f260a94eeb879ec0a43b5284c6
I20260503 13:57:06.677503 8378 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260503 13:57:06.693475 8378 catalog_manager.cc:1380] Generated new certificate authority record
I20260503 13:57:06.695511 8378 catalog_manager.cc:1514] Loading token signing keys...
I20260503 13:57:06.722445 8378 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2: Generated new TSK 0
I20260503 13:57:06.723452 8378 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260503 13:57:06.729010 8299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260503 13:57:06.738178 8388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:57:06.740345 8389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:57:06.742733 8391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260503 13:57:06.742866 8299 server_base.cc:1061] running on GCE node
I20260503 13:57:06.743656 8299 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260503 13:57:06.743865 8299 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260503 13:57:06.744033 8299 hybrid_clock.cc:648] HybridClock initialized: now 1777816626744016 us; error 0 us; skew 500 ppm
I20260503 13:57:06.744611 8299 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260503 13:57:06.748649 8299 webserver.cc:492] Webserver started at http://127.8.26.193:39639/ using document root <none> and password file <none>
I20260503 13:57:06.749185 8299 fs_manager.cc:362] Metadata directory not provided
I20260503 13:57:06.749380 8299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260503 13:57:06.749635 8299 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260503 13:57:06.750777 8299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-0-root/instance:
uuid: "1efe002803924b75971d7948c1675b12"
format_stamp: "Formatted at 2026-05-03 13:57:06 on dist-test-slave-jcqq"
I20260503 13:57:06.756105 8299 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260503 13:57:06.760000 8396 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:06.760808 8299 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260503 13:57:06.761080 8299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-0-root
uuid: "1efe002803924b75971d7948c1675b12"
format_stamp: "Formatted at 2026-05-03 13:57:06 on dist-test-slave-jcqq"
I20260503 13:57:06.761353 8299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260503 13:57:06.778196 8299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260503 13:57:06.779502 8299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260503 13:57:06.781991 8299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260503 13:57:06.786113 8299 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260503 13:57:06.786337 8299 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:06.786579 8299 ts_tablet_manager.cc:616] Registered 0 tablets
I20260503 13:57:06.786751 8299 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:06.878955 8299 rpc_server.cc:307] RPC server started. Bound to: 127.8.26.193:46633
I20260503 13:57:06.879036 8458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.26.193:46633 every 8 connection(s)
I20260503 13:57:06.885663 8299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260503 13:57:06.893867 8463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:57:06.899066 8464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260503 13:57:06.903071 8299 server_base.cc:1061] running on GCE node
W20260503 13:57:06.904642 8466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260503 13:57:06.905727 8299 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260503 13:57:06.906010 8299 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260503 13:57:06.906212 8299 hybrid_clock.cc:648] HybridClock initialized: now 1777816626906196 us; error 0 us; skew 500 ppm
I20260503 13:57:06.906649 8459 heartbeater.cc:344] Connected to a master server at 127.8.26.254:44623
I20260503 13:57:06.906924 8299 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260503 13:57:06.907054 8459 heartbeater.cc:461] Registering TS with master...
I20260503 13:57:06.908064 8459 heartbeater.cc:507] Master 127.8.26.254:44623 requested a full tablet report, sending...
I20260503 13:57:06.910842 8331 ts_manager.cc:194] Registered new tserver with Master: 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633)
I20260503 13:57:06.911012 8299 webserver.cc:492] Webserver started at http://127.8.26.194:36581/ using document root <none> and password file <none>
I20260503 13:57:06.911696 8299 fs_manager.cc:362] Metadata directory not provided
I20260503 13:57:06.911944 8299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260503 13:57:06.912281 8299 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260503 13:57:06.912982 8331 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36670
I20260503 13:57:06.913722 8299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-1-root/instance:
uuid: "a140d8dd75a24d018e082b3ba9603de7"
format_stamp: "Formatted at 2026-05-03 13:57:06 on dist-test-slave-jcqq"
I20260503 13:57:06.920296 8299 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260503 13:57:06.925041 8471 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:06.925806 8299 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260503 13:57:06.926126 8299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-1-root
uuid: "a140d8dd75a24d018e082b3ba9603de7"
format_stamp: "Formatted at 2026-05-03 13:57:06 on dist-test-slave-jcqq"
I20260503 13:57:06.926453 8299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260503 13:57:06.953658 8299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260503 13:57:06.955231 8299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260503 13:57:06.957624 8299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260503 13:57:06.961035 8299 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260503 13:57:06.961290 8299 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:06.961556 8299 ts_tablet_manager.cc:616] Registered 0 tablets
I20260503 13:57:06.961781 8299 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:07.043519 8299 rpc_server.cc:307] RPC server started. Bound to: 127.8.26.194:36889
I20260503 13:57:07.043691 8533 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.26.194:36889 every 8 connection(s)
I20260503 13:57:07.051021 8299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260503 13:57:07.058513 8537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:57:07.061398 8538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260503 13:57:07.063174 8534 heartbeater.cc:344] Connected to a master server at 127.8.26.254:44623
I20260503 13:57:07.063565 8534 heartbeater.cc:461] Registering TS with master...
I20260503 13:57:07.064494 8534 heartbeater.cc:507] Master 127.8.26.254:44623 requested a full tablet report, sending...
I20260503 13:57:07.066768 8331 ts_manager.cc:194] Registered new tserver with Master: a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889)
I20260503 13:57:07.068495 8331 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36686
W20260503 13:57:07.073441 8540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260503 13:57:07.074152 8299 server_base.cc:1061] running on GCE node
I20260503 13:57:07.074878 8299 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260503 13:57:07.075057 8299 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260503 13:57:07.075210 8299 hybrid_clock.cc:648] HybridClock initialized: now 1777816627075192 us; error 0 us; skew 500 ppm
I20260503 13:57:07.075687 8299 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260503 13:57:07.078343 8299 webserver.cc:492] Webserver started at http://127.8.26.195:33941/ using document root <none> and password file <none>
I20260503 13:57:07.078783 8299 fs_manager.cc:362] Metadata directory not provided
I20260503 13:57:07.078950 8299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260503 13:57:07.079185 8299 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260503 13:57:07.080384 8299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-2-root/instance:
uuid: "ff09527976954240a2b7e45977386f61"
format_stamp: "Formatted at 2026-05-03 13:57:07 on dist-test-slave-jcqq"
I20260503 13:57:07.084707 8299 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260503 13:57:07.087873 8545 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:07.088670 8299 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260503 13:57:07.088989 8299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-2-root
uuid: "ff09527976954240a2b7e45977386f61"
format_stamp: "Formatted at 2026-05-03 13:57:07 on dist-test-slave-jcqq"
I20260503 13:57:07.089332 8299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777816625017654-8299-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260503 13:57:07.104967 8299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260503 13:57:07.106493 8299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260503 13:57:07.108938 8299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260503 13:57:07.112356 8299 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260503 13:57:07.112627 8299 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260503 13:57:07.112908 8299 ts_tablet_manager.cc:616] Registered 0 tablets
I20260503 13:57:07.113122 8299 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:07.193001 8299 rpc_server.cc:307] RPC server started. Bound to: 127.8.26.195:40111
I20260503 13:57:07.193059 8607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.26.195:40111 every 8 connection(s)
I20260503 13:57:07.209712 8608 heartbeater.cc:344] Connected to a master server at 127.8.26.254:44623
I20260503 13:57:07.210078 8608 heartbeater.cc:461] Registering TS with master...
I20260503 13:57:07.210851 8608 heartbeater.cc:507] Master 127.8.26.254:44623 requested a full tablet report, sending...
I20260503 13:57:07.212638 8331 ts_manager.cc:194] Registered new tserver with Master: ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:07.212800 8299 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014582442s
I20260503 13:57:07.213869 8331 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36690
I20260503 13:57:07.259238 8331 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36700:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260503 13:57:07.262657 8331 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260503 13:57:07.356364 8571 tablet_service.cc:1511] Processing CreateTablet for tablet cbd326985c85427392b9ecb990c3d888 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260503 13:57:07.358017 8571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd326985c85427392b9ecb990c3d888. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.358063 8567 tablet_service.cc:1511] Processing CreateTablet for tablet 1fe7524a546346368ea1f28ee97c221a (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260503 13:57:07.359391 8567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fe7524a546346368ea1f28ee97c221a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.361650 8568 tablet_service.cc:1511] Processing CreateTablet for tablet 566555790b4d4399bbc0571468916a89 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260503 13:57:07.361811 8570 tablet_service.cc:1511] Processing CreateTablet for tablet 9031aeee0f614639943edc1aa0067412 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260503 13:57:07.362875 8568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 566555790b4d4399bbc0571468916a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.363533 8570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9031aeee0f614639943edc1aa0067412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.365125 8573 tablet_service.cc:1511] Processing CreateTablet for tablet a900eebd056343a89cef507a78e29e55 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260503 13:57:07.366331 8573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a900eebd056343a89cef507a78e29e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.367833 8572 tablet_service.cc:1511] Processing CreateTablet for tablet f2a6dc6f02264540ac18b8a431c42e4a (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260503 13:57:07.369143 8572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2a6dc6f02264540ac18b8a431c42e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.373732 8569 tablet_service.cc:1511] Processing CreateTablet for tablet 941dbfe411bf460e9a1563b4b946f9a7 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260503 13:57:07.374994 8569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 941dbfe411bf460e9a1563b4b946f9a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.378258 8566 tablet_service.cc:1511] Processing CreateTablet for tablet d7caf0031e354b8ca998e7640505cb9b (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260503 13:57:07.379451 8566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7caf0031e354b8ca998e7640505cb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.381984 8565 tablet_service.cc:1511] Processing CreateTablet for tablet 76a4ada557094372a9d2a6c9b9c24cd6 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260503 13:57:07.385121 8565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a4ada557094372a9d2a6c9b9c24cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.399504 8422 tablet_service.cc:1511] Processing CreateTablet for tablet f2a6dc6f02264540ac18b8a431c42e4a (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260503 13:57:07.398262 8424 tablet_service.cc:1511] Processing CreateTablet for tablet a900eebd056343a89cef507a78e29e55 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260503 13:57:07.402544 8422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2a6dc6f02264540ac18b8a431c42e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.403512 8424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a900eebd056343a89cef507a78e29e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.410781 8423 tablet_service.cc:1511] Processing CreateTablet for tablet cbd326985c85427392b9ecb990c3d888 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260503 13:57:07.412339 8423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd326985c85427392b9ecb990c3d888. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.428579 8498 tablet_service.cc:1511] Processing CreateTablet for tablet f2a6dc6f02264540ac18b8a431c42e4a (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260503 13:57:07.429900 8498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2a6dc6f02264540ac18b8a431c42e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.439425 8499 tablet_service.cc:1511] Processing CreateTablet for tablet a900eebd056343a89cef507a78e29e55 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260503 13:57:07.440380 8421 tablet_service.cc:1511] Processing CreateTablet for tablet 9031aeee0f614639943edc1aa0067412 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260503 13:57:07.446005 8419 tablet_service.cc:1511] Processing CreateTablet for tablet 566555790b4d4399bbc0571468916a89 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260503 13:57:07.444504 8420 tablet_service.cc:1511] Processing CreateTablet for tablet 941dbfe411bf460e9a1563b4b946f9a7 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260503 13:57:07.455333 8421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9031aeee0f614639943edc1aa0067412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.457683 8495 tablet_service.cc:1511] Processing CreateTablet for tablet 941dbfe411bf460e9a1563b4b946f9a7 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260503 13:57:07.457060 8417 tablet_service.cc:1511] Processing CreateTablet for tablet d7caf0031e354b8ca998e7640505cb9b (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260503 13:57:07.458983 8499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a900eebd056343a89cef507a78e29e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.451295 8497 tablet_service.cc:1511] Processing CreateTablet for tablet cbd326985c85427392b9ecb990c3d888 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260503 13:57:07.455344 8496 tablet_service.cc:1511] Processing CreateTablet for tablet 9031aeee0f614639943edc1aa0067412 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260503 13:57:07.464766 8420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 941dbfe411bf460e9a1563b4b946f9a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.467350 8625 tablet_bootstrap.cc:492] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61: Bootstrap starting.
I20260503 13:57:07.473206 8495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 941dbfe411bf460e9a1563b4b946f9a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.471148 8418 tablet_service.cc:1511] Processing CreateTablet for tablet 1fe7524a546346368ea1f28ee97c221a (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260503 13:57:07.475847 8416 tablet_service.cc:1511] Processing CreateTablet for tablet 76a4ada557094372a9d2a6c9b9c24cd6 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260503 13:57:07.479924 8417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7caf0031e354b8ca998e7640505cb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.479614 8494 tablet_service.cc:1511] Processing CreateTablet for tablet 566555790b4d4399bbc0571468916a89 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260503 13:57:07.478667 8492 tablet_service.cc:1511] Processing CreateTablet for tablet d7caf0031e354b8ca998e7640505cb9b (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260503 13:57:07.480283 8491 tablet_service.cc:1511] Processing CreateTablet for tablet 76a4ada557094372a9d2a6c9b9c24cd6 (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260503 13:57:07.485954 8493 tablet_service.cc:1511] Processing CreateTablet for tablet 1fe7524a546346368ea1f28ee97c221a (DEFAULT_TABLE table=test-workload [id=b14971766ff944f287012c76238a0862]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260503 13:57:07.488161 8497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd326985c85427392b9ecb990c3d888. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.490535 8494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 566555790b4d4399bbc0571468916a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.497567 8491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a4ada557094372a9d2a6c9b9c24cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.502535 8492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7caf0031e354b8ca998e7640505cb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.502970 8419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 566555790b4d4399bbc0571468916a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.505798 8496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9031aeee0f614639943edc1aa0067412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.513840 8418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fe7524a546346368ea1f28ee97c221a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.521230 8493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fe7524a546346368ea1f28ee97c221a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.549045 8416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a4ada557094372a9d2a6c9b9c24cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:07.593269 8626 tablet_bootstrap.cc:492] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7: Bootstrap starting.
W20260503 13:57:07.595985 8318 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.8.26.195:40111, user_credentials={real_user=slave}} blocked reactor thread for 57138.9us
I20260503 13:57:07.605351 8626 tablet_bootstrap.cc:654] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:07.626762 8627 tablet_bootstrap.cc:492] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12: Bootstrap starting.
I20260503 13:57:07.629655 8625 tablet_bootstrap.cc:654] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61: Neither blocks nor log segments found. Creating new log.
W20260503 13:57:07.653656 8379 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:57:07.656965 8626 tablet_bootstrap.cc:492] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7: No bootstrap required, opened a new log
I20260503 13:57:07.657423 8626 ts_tablet_manager.cc:1403] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7: Time spent bootstrapping tablet: real 0.130s user 0.012s sys 0.015s
I20260503 13:57:07.667347 8627 tablet_bootstrap.cc:654] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:07.675374 8626 raft_consensus.cc:359] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.676136 8626 raft_consensus.cc:385] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:07.676436 8626 raft_consensus.cc:740] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Initialized, Role: FOLLOWER
I20260503 13:57:07.680649 8626 consensus_queue.cc:260] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.684419 8625 tablet_bootstrap.cc:492] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61: No bootstrap required, opened a new log
I20260503 13:57:07.685014 8625 ts_tablet_manager.cc:1403] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61: Time spent bootstrapping tablet: real 0.240s user 0.002s sys 0.014s
I20260503 13:57:07.688628 8625 raft_consensus.cc:359] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:07.689361 8625 raft_consensus.cc:385] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:07.689644 8625 raft_consensus.cc:740] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff09527976954240a2b7e45977386f61, State: Initialized, Role: FOLLOWER
I20260503 13:57:07.690646 8625 consensus_queue.cc:260] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:07.695652 8627 tablet_bootstrap.cc:492] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12: No bootstrap required, opened a new log
I20260503 13:57:07.696138 8627 ts_tablet_manager.cc:1403] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.009s
I20260503 13:57:07.697824 8608 heartbeater.cc:499] Master 127.8.26.254:44623 was elected leader, sending a full tablet report...
I20260503 13:57:07.698205 8625 ts_tablet_manager.cc:1434] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260503 13:57:07.699064 8625 tablet_bootstrap.cc:492] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61: Bootstrap starting.
I20260503 13:57:07.699568 8627 raft_consensus.cc:359] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:07.700254 8627 raft_consensus.cc:385] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:07.700525 8627 raft_consensus.cc:740] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Initialized, Role: FOLLOWER
W20260503 13:57:07.700944 8609 tablet.cc:2404] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260503 13:57:07.702334 8627 consensus_queue.cc:260] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:07.703090 8626 ts_tablet_manager.cc:1434] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7: Time spent starting tablet: real 0.045s user 0.007s sys 0.002s
I20260503 13:57:07.703063 8534 heartbeater.cc:499] Master 127.8.26.254:44623 was elected leader, sending a full tablet report...
I20260503 13:57:07.706557 8625 tablet_bootstrap.cc:654] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:07.712001 8626 tablet_bootstrap.cc:492] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7: Bootstrap starting.
I20260503 13:57:07.718564 8459 heartbeater.cc:499] Master 127.8.26.254:44623 was elected leader, sending a full tablet report...
I20260503 13:57:07.718580 8627 ts_tablet_manager.cc:1434] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12: Time spent starting tablet: real 0.022s user 0.000s sys 0.022s
I20260503 13:57:07.720484 8627 tablet_bootstrap.cc:492] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12: Bootstrap starting.
I20260503 13:57:07.726976 8626 tablet_bootstrap.cc:654] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:07.727448 8625 tablet_bootstrap.cc:492] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61: No bootstrap required, opened a new log
I20260503 13:57:07.727933 8625 ts_tablet_manager.cc:1403] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.001s
I20260503 13:57:07.731009 8625 raft_consensus.cc:359] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.731734 8625 raft_consensus.cc:385] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:07.732056 8625 raft_consensus.cc:740] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff09527976954240a2b7e45977386f61, State: Initialized, Role: FOLLOWER
I20260503 13:57:07.732791 8625 consensus_queue.cc:260] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.733724 8627 tablet_bootstrap.cc:654] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:07.735316 8625 ts_tablet_manager.cc:1434] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260503 13:57:07.736296 8625 tablet_bootstrap.cc:492] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61: Bootstrap starting.
I20260503 13:57:07.743088 8625 tablet_bootstrap.cc:654] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:07.747838 8627 tablet_bootstrap.cc:492] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12: No bootstrap required, opened a new log
I20260503 13:57:07.749348 8627 ts_tablet_manager.cc:1403] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.004s
I20260503 13:57:07.753140 8627 raft_consensus.cc:359] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.757509 8627 raft_consensus.cc:385] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:07.756330 8625 tablet_bootstrap.cc:492] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61: No bootstrap required, opened a new log
I20260503 13:57:07.758247 8627 raft_consensus.cc:740] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Initialized, Role: FOLLOWER
I20260503 13:57:07.758636 8625 ts_tablet_manager.cc:1403] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.007s
I20260503 13:57:07.761648 8625 raft_consensus.cc:359] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.762324 8625 raft_consensus.cc:385] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:07.762689 8625 raft_consensus.cc:740] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff09527976954240a2b7e45977386f61, State: Initialized, Role: FOLLOWER
I20260503 13:57:07.762501 8627 consensus_queue.cc:260] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.763345 8625 consensus_queue.cc:260] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.765075 8627 ts_tablet_manager.cc:1434] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12: Time spent starting tablet: real 0.015s user 0.001s sys 0.005s
I20260503 13:57:07.765594 8625 ts_tablet_manager.cc:1434] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260503 13:57:07.766072 8627 tablet_bootstrap.cc:492] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12: Bootstrap starting.
I20260503 13:57:07.766434 8625 tablet_bootstrap.cc:492] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61: Bootstrap starting.
I20260503 13:57:07.770567 8626 tablet_bootstrap.cc:492] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7: No bootstrap required, opened a new log
I20260503 13:57:07.771009 8626 ts_tablet_manager.cc:1403] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7: Time spent bootstrapping tablet: real 0.059s user 0.017s sys 0.008s
I20260503 13:57:07.772688 8625 tablet_bootstrap.cc:654] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:07.773746 8626 raft_consensus.cc:359] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.774418 8626 raft_consensus.cc:385] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:07.774684 8626 raft_consensus.cc:740] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Initialized, Role: FOLLOWER
I20260503 13:57:07.775341 8626 consensus_queue.cc:260] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.777940 8626 ts_tablet_manager.cc:1434] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260503 13:57:07.781344 8626 tablet_bootstrap.cc:492] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7: Bootstrap starting.
I20260503 13:57:07.787511 8626 tablet_bootstrap.cc:654] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:07.787652 8625 tablet_bootstrap.cc:492] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61: No bootstrap required, opened a new log
I20260503 13:57:07.788535 8627 tablet_bootstrap.cc:654] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:07.788941 8625 ts_tablet_manager.cc:1403] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.016s
I20260503 13:57:07.791707 8625 raft_consensus.cc:359] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.792389 8625 raft_consensus.cc:385] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:07.792701 8625 raft_consensus.cc:740] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff09527976954240a2b7e45977386f61, State: Initialized, Role: FOLLOWER
I20260503 13:57:07.793360 8625 consensus_queue.cc:260] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.795826 8625 ts_tablet_manager.cc:1434] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260503 13:57:07.796795 8625 tablet_bootstrap.cc:492] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61: Bootstrap starting.
I20260503 13:57:07.803126 8625 tablet_bootstrap.cc:654] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:07.814267 8632 raft_consensus.cc:493] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:07.814937 8627 tablet_bootstrap.cc:492] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12: No bootstrap required, opened a new log
I20260503 13:57:07.814699 8632 raft_consensus.cc:515] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:07.815387 8627 ts_tablet_manager.cc:1403] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12: Time spent bootstrapping tablet: real 0.049s user 0.017s sys 0.010s
I20260503 13:57:07.817595 8632 leader_election.cc:290] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633)
I20260503 13:57:07.819232 8625 tablet_bootstrap.cc:492] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61: No bootstrap required, opened a new log
I20260503 13:57:07.819644 8625 ts_tablet_manager.cc:1403] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.004s
I20260503 13:57:07.820597 8627 raft_consensus.cc:359] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.822649 8627 raft_consensus.cc:385] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:07.823350 8627 raft_consensus.cc:740] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Initialized, Role: FOLLOWER
I20260503 13:57:07.823038 8625 raft_consensus.cc:359] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.824832 8625 raft_consensus.cc:385] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:07.825141 8625 raft_consensus.cc:740] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff09527976954240a2b7e45977386f61, State: Initialized, Role: FOLLOWER
I20260503 13:57:07.824667 8627 consensus_queue.cc:260] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.825860 8625 consensus_queue.cc:260] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.828375 8627 ts_tablet_manager.cc:1434] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12: Time spent starting tablet: real 0.013s user 0.002s sys 0.005s
I20260503 13:57:07.828505 8625 ts_tablet_manager.cc:1434] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260503 13:57:07.830742 8625 tablet_bootstrap.cc:492] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61: Bootstrap starting.
I20260503 13:57:07.830763 8627 tablet_bootstrap.cc:492] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12: Bootstrap starting.
I20260503 13:57:07.837364 8625 tablet_bootstrap.cc:654] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:07.842926 8631 raft_consensus.cc:493] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:07.843382 8631 raft_consensus.cc:515] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.843782 8626 tablet_bootstrap.cc:492] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7: No bootstrap required, opened a new log
I20260503 13:57:07.845366 8626 ts_tablet_manager.cc:1403] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.010s
I20260503 13:57:07.846184 8627 tablet_bootstrap.cc:654] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:07.846630 8631 leader_election.cc:290] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:07.874747 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a900eebd056343a89cef507a78e29e55" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12" is_pre_election: true
I20260503 13:57:07.875555 8434 raft_consensus.cc:2468] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 0.
I20260503 13:57:07.875967 8626 raft_consensus.cc:359] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.876722 8626 raft_consensus.cc:385] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:07.877049 8626 raft_consensus.cc:740] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Initialized, Role: FOLLOWER
I20260503 13:57:07.877142 8549 leader_election.cc:304] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:07.878077 8632 raft_consensus.cc:2804] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:07.879096 8632 raft_consensus.cc:493] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:07.879426 8632 raft_consensus.cc:3060] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:07.884662 8633 raft_consensus.cc:493] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:07.879637 8626 consensus_queue.cc:260] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.885848 8633 raft_consensus.cc:515] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.888381 8626 ts_tablet_manager.cc:1434] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7: Time spent starting tablet: real 0.043s user 0.007s sys 0.000s
I20260503 13:57:07.889971 8625 tablet_bootstrap.cc:492] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61: No bootstrap required, opened a new log
I20260503 13:57:07.890497 8633 leader_election.cc:290] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:07.892779 8625 ts_tablet_manager.cc:1403] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61: Time spent bootstrapping tablet: real 0.062s user 0.018s sys 0.011s
I20260503 13:57:07.892369 8626 tablet_bootstrap.cc:492] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7: Bootstrap starting.
I20260503 13:57:07.895444 8627 tablet_bootstrap.cc:492] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12: No bootstrap required, opened a new log
I20260503 13:57:07.895907 8627 ts_tablet_manager.cc:1403] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.004s
I20260503 13:57:07.898813 8627 raft_consensus.cc:359] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.899513 8627 raft_consensus.cc:385] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:07.899806 8627 raft_consensus.cc:740] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Initialized, Role: FOLLOWER
I20260503 13:57:07.900470 8627 consensus_queue.cc:260] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.905524 8632 raft_consensus.cc:515] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:07.912272 8625 raft_consensus.cc:359] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.912966 8625 raft_consensus.cc:385] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:07.913270 8625 raft_consensus.cc:740] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff09527976954240a2b7e45977386f61, State: Initialized, Role: FOLLOWER
I20260503 13:57:07.914922 8625 consensus_queue.cc:260] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.916342 8626 tablet_bootstrap.cc:654] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:07.922288 8632 leader_election.cc:290] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 election: Requested vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633)
I20260503 13:57:07.923259 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a900eebd056343a89cef507a78e29e55" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:07.923769 8434 raft_consensus.cc:3060] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:07.923867 8627 ts_tablet_manager.cc:1434] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12: Time spent starting tablet: real 0.028s user 0.006s sys 0.002s
I20260503 13:57:07.928996 8627 tablet_bootstrap.cc:492] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12: Bootstrap starting.
I20260503 13:57:07.942286 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a900eebd056343a89cef507a78e29e55" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7" is_pre_election: true
I20260503 13:57:07.943800 8626 tablet_bootstrap.cc:492] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7: No bootstrap required, opened a new log
I20260503 13:57:07.943833 8434 raft_consensus.cc:2468] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 1.
I20260503 13:57:07.945210 8549 leader_election.cc:304] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:07.944348 8626 ts_tablet_manager.cc:1403] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.003s
I20260503 13:57:07.945636 8625 ts_tablet_manager.cc:1434] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61: Time spent starting tablet: real 0.053s user 0.005s sys 0.006s
I20260503 13:57:07.948230 8653 raft_consensus.cc:2804] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:07.948751 8625 tablet_bootstrap.cc:492] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61: Bootstrap starting.
I20260503 13:57:07.949414 8632 raft_consensus.cc:493] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:07.950805 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a900eebd056343a89cef507a78e29e55" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:07.950374 8632 raft_consensus.cc:515] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.954950 8626 raft_consensus.cc:359] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.955608 8626 raft_consensus.cc:385] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:07.955884 8626 raft_consensus.cc:740] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Initialized, Role: FOLLOWER
W20260503 13:57:07.953879 8547 leader_election.cc:343] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:07.948818 8653 raft_consensus.cc:697] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 1 LEADER]: Becoming Leader. State: Replica: ff09527976954240a2b7e45977386f61, State: Running, Role: LEADER
W20260503 13:57:07.958320 8547 leader_election.cc:343] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:07.958647 8653 consensus_queue.cc:237] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:07.960332 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12" is_pre_election: true
I20260503 13:57:07.960922 8434 raft_consensus.cc:2468] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 0.
I20260503 13:57:07.961975 8549 leader_election.cc:304] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:07.957127 8632 leader_election.cc:290] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889)
I20260503 13:57:07.956529 8626 consensus_queue.cc:260] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:07.963227 8632 raft_consensus.cc:2804] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:07.964385 8632 raft_consensus.cc:493] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:07.964454 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7" is_pre_election: true
I20260503 13:57:07.968688 8509 raft_consensus.cc:2468] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 0.
I20260503 13:57:07.964854 8632 raft_consensus.cc:3060] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:07.975822 8626 ts_tablet_manager.cc:1434] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7: Time spent starting tablet: real 0.030s user 0.002s sys 0.005s
I20260503 13:57:07.979391 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7" is_pre_election: true
I20260503 13:57:07.980014 8509 raft_consensus.cc:2468] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 0.
I20260503 13:57:07.980000 8626 tablet_bootstrap.cc:492] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7: Bootstrap starting.
I20260503 13:57:07.981184 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12" is_pre_election: true
I20260503 13:57:07.982787 8434 raft_consensus.cc:2468] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 0.
I20260503 13:57:07.984242 8398 leader_election.cc:304] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7; no voters:
I20260503 13:57:07.986996 8633 raft_consensus.cc:2804] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:07.987351 8633 raft_consensus.cc:493] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:07.987663 8633 raft_consensus.cc:3060] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:07.988217 8475 leader_election.cc:304] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7; no voters:
I20260503 13:57:07.988229 8627 tablet_bootstrap.cc:654] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:07.989969 8631 raft_consensus.cc:2804] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:07.992501 8631 raft_consensus.cc:493] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:07.995553 8631 raft_consensus.cc:3060] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.004853 8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61" is_pre_election: true
I20260503 13:57:08.008468 8633 raft_consensus.cc:515] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.009316 8631 raft_consensus.cc:515] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.010336 8633 leader_election.cc:290] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 election: Requested vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.010672 8655 raft_consensus.cc:493] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:08.011050 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:08.011780 8509 raft_consensus.cc:3060] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.012588 8632 raft_consensus.cc:515] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.014555 8583 raft_consensus.cc:2393] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a140d8dd75a24d018e082b3ba9603de7 in current term 1: Already voted for candidate ff09527976954240a2b7e45977386f61 in this term.
I20260503 13:57:08.011102 8655 raft_consensus.cc:515] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.017048 8631 leader_election.cc:290] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.018399 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:08.020469 8434 raft_consensus.cc:3060] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.023900 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:08.020946 8632 leader_election.cc:290] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889)
I20260503 13:57:08.024907 8508 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:08.026001 8631 raft_consensus.cc:493] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:08.027511 8631 raft_consensus.cc:515] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.031697 8508 raft_consensus.cc:2393] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ff09527976954240a2b7e45977386f61 in current term 1: Already voted for candidate a140d8dd75a24d018e082b3ba9603de7 in this term.
I20260503 13:57:08.031816 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61" is_pre_election: true
I20260503 13:57:08.032218 8626 tablet_bootstrap.cc:654] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:08.032959 8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:08.034771 8433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12"
W20260503 13:57:08.036321 8400 leader_election.cc:343] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff09527976954240a2b7e45977386f61 (127.8.26.195:40111): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:57:08.037186 8400 leader_election.cc:343] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff09527976954240a2b7e45977386f61 (127.8.26.195:40111): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:08.037388 8631 leader_election.cc:290] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.035135 8625 tablet_bootstrap.cc:654] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:08.038580 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61" is_pre_election: true
I20260503 13:57:08.038686 8433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12" is_pre_election: true
I20260503 13:57:08.039422 8582 raft_consensus.cc:2468] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 0.
W20260503 13:57:08.040117 8475 leader_election.cc:343] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:08.041371 8475 leader_election.cc:304] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a140d8dd75a24d018e082b3ba9603de7, ff09527976954240a2b7e45977386f61; no voters: 1efe002803924b75971d7948c1675b12
I20260503 13:57:08.043469 8627 tablet_bootstrap.cc:492] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12: No bootstrap required, opened a new log
I20260503 13:57:08.043903 8627 ts_tablet_manager.cc:1403] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12: Time spent bootstrapping tablet: real 0.115s user 0.005s sys 0.014s
I20260503 13:57:08.042754 8631 raft_consensus.cc:2804] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:08.045346 8434 raft_consensus.cc:2468] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 1.
I20260503 13:57:08.047271 8509 raft_consensus.cc:2468] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 1.
I20260503 13:57:08.047299 8475 leader_election.cc:304] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7; no voters: ff09527976954240a2b7e45977386f61
I20260503 13:57:08.047107 8627 raft_consensus.cc:359] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.047981 8627 raft_consensus.cc:385] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:08.047366 8631 raft_consensus.cc:493] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:08.048950 8627 raft_consensus.cc:740] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Initialized, Role: FOLLOWER
I20260503 13:57:08.044745 8549 leader_election.cc:304] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff09527976954240a2b7e45977386f61; no voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7
I20260503 13:57:08.049908 8627 consensus_queue.cc:260] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.052474 8655 leader_election.cc:290] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889)
I20260503 13:57:08.053246 8632 raft_consensus.cc:2749] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260503 13:57:08.052613 8398 leader_election.cc:304] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7; no voters: ff09527976954240a2b7e45977386f61
I20260503 13:57:08.049203 8631 raft_consensus.cc:3060] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.060212 8633 raft_consensus.cc:2804] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:08.060060 8627 ts_tablet_manager.cc:1434] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12: Time spent starting tablet: real 0.016s user 0.003s sys 0.004s
I20260503 13:57:08.065615 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "941dbfe411bf460e9a1563b4b946f9a7" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12" is_pre_election: true
I20260503 13:57:08.066412 8627 tablet_bootstrap.cc:492] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12: Bootstrap starting.
W20260503 13:57:08.066923 8549 leader_election.cc:343] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:08.060693 8633 raft_consensus.cc:697] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Becoming Leader. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:08.065652 8658 raft_consensus.cc:2804] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:08.059767 8330 catalog_manager.cc:5671] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff09527976954240a2b7e45977386f61 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "ff09527976954240a2b7e45977386f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:08.074913 8625 tablet_bootstrap.cc:492] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61: No bootstrap required, opened a new log
I20260503 13:57:08.075351 8625 ts_tablet_manager.cc:1403] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61: Time spent bootstrapping tablet: real 0.127s user 0.013s sys 0.006s
I20260503 13:57:08.077595 8626 tablet_bootstrap.cc:492] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7: No bootstrap required, opened a new log
I20260503 13:57:08.078029 8626 ts_tablet_manager.cc:1403] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7: Time spent bootstrapping tablet: real 0.100s user 0.016s sys 0.002s
I20260503 13:57:08.073055 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "941dbfe411bf460e9a1563b4b946f9a7" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7" is_pre_election: true
W20260503 13:57:08.080408 8547 leader_election.cc:343] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:08.078259 8625 raft_consensus.cc:359] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.081418 8547 leader_election.cc:304] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff09527976954240a2b7e45977386f61; no voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7
I20260503 13:57:08.082945 8631 raft_consensus.cc:515] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.083346 8632 raft_consensus.cc:2749] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:57:08.081970 8625 raft_consensus.cc:385] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:08.074579 8633 consensus_queue.cc:237] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.087757 8627 tablet_bootstrap.cc:654] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:08.089260 8626 raft_consensus.cc:359] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:08.090241 8631 leader_election.cc:290] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.084113 8625 raft_consensus.cc:740] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff09527976954240a2b7e45977386f61, State: Initialized, Role: FOLLOWER
I20260503 13:57:08.091858 8632 raft_consensus.cc:493] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:08.091995 8625 consensus_queue.cc:260] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.092357 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:08.092304 8632 raft_consensus.cc:515] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.094192 8632 leader_election.cc:290] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889)
I20260503 13:57:08.094478 8631 raft_consensus.cc:493] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:08.094596 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:08.095122 8582 raft_consensus.cc:3060] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.094925 8631 raft_consensus.cc:515] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.094158 8625 ts_tablet_manager.cc:1434] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260503 13:57:08.090875 8626 raft_consensus.cc:385] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:08.091225 8658 raft_consensus.cc:697] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Becoming Leader. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
W20260503 13:57:08.097380 8475 leader_election.cc:343] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:08.098246 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566555790b4d4399bbc0571468916a89" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12" is_pre_election: true
I20260503 13:57:08.100776 8626 raft_consensus.cc:740] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Initialized, Role: FOLLOWER
I20260503 13:57:08.101352 8658 consensus_queue.cc:237] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.103155 8434 raft_consensus.cc:2468] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 0.
I20260503 13:57:08.103514 8626 consensus_queue.cc:260] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:08.103969 8433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031aeee0f614639943edc1aa0067412" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12" is_pre_election: true
I20260503 13:57:08.104149 8549 leader_election.cc:304] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:08.104915 8632 raft_consensus.cc:2804] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:08.105397 8632 raft_consensus.cc:493] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:08.105573 8626 ts_tablet_manager.cc:1434] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7: Time spent starting tablet: real 0.027s user 0.004s sys 0.004s
I20260503 13:57:08.104904 8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031aeee0f614639943edc1aa0067412" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61" is_pre_election: true
I20260503 13:57:08.097463 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566555790b4d4399bbc0571468916a89" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7" is_pre_election: true
I20260503 13:57:08.105712 8632 raft_consensus.cc:3060] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.102043 8631 leader_election.cc:290] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
W20260503 13:57:08.106706 8475 leader_election.cc:343] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:08.106779 8581 raft_consensus.cc:2468] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 0.
W20260503 13:57:08.107712 8547 leader_election.cc:343] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:08.110961 8626 tablet_bootstrap.cc:492] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7: Bootstrap starting.
I20260503 13:57:08.113978 8582 raft_consensus.cc:2468] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 1.
I20260503 13:57:08.117573 8632 raft_consensus.cc:515] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.110004 8475 leader_election.cc:304] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a140d8dd75a24d018e082b3ba9603de7, ff09527976954240a2b7e45977386f61; no voters: 1efe002803924b75971d7948c1675b12
I20260503 13:57:08.123553 8626 tablet_bootstrap.cc:654] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:08.125691 8662 raft_consensus.cc:2804] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:08.126125 8662 raft_consensus.cc:493] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:08.126507 8662 raft_consensus.cc:3060] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.126801 8433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566555790b4d4399bbc0571468916a89" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:08.127772 8433 raft_consensus.cc:3060] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.127554 8625 tablet_bootstrap.cc:492] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61: Bootstrap starting.
I20260503 13:57:08.124590 8475 leader_election.cc:304] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a140d8dd75a24d018e082b3ba9603de7, ff09527976954240a2b7e45977386f61; no voters: 1efe002803924b75971d7948c1675b12
I20260503 13:57:08.130586 8632 leader_election.cc:290] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889)
I20260503 13:57:08.133136 8631 raft_consensus.cc:2804] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:08.134263 8631 raft_consensus.cc:697] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Becoming Leader. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:08.132298 8331 catalog_manager.cc:5671] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1efe002803924b75971d7948c1675b12 (127.8.26.193). New cstate: current_term: 1 leader_uuid: "1efe002803924b75971d7948c1675b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:08.135076 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566555790b4d4399bbc0571468916a89" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
W20260503 13:57:08.136984 8547 leader_election.cc:343] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:08.136353 8631 consensus_queue.cc:237] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.141629 8433 raft_consensus.cc:2468] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 1.
I20260503 13:57:08.143491 8549 leader_election.cc:304] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61; no voters: a140d8dd75a24d018e082b3ba9603de7
I20260503 13:57:08.152683 8632 raft_consensus.cc:2804] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:08.158784 8632 raft_consensus.cc:697] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 1 LEADER]: Becoming Leader. State: Replica: ff09527976954240a2b7e45977386f61, State: Running, Role: LEADER
I20260503 13:57:08.159564 8632 consensus_queue.cc:237] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.173537 8627 tablet_bootstrap.cc:492] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12: No bootstrap required, opened a new log
I20260503 13:57:08.174057 8627 ts_tablet_manager.cc:1403] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12: Time spent bootstrapping tablet: real 0.108s user 0.011s sys 0.007s
I20260503 13:57:08.181699 8625 tablet_bootstrap.cc:654] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:08.172224 8662 raft_consensus.cc:515] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.195262 8433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031aeee0f614639943edc1aa0067412" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:08.196460 8662 leader_election.cc:290] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
W20260503 13:57:08.196700 8475 leader_election.cc:343] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:08.197602 8626 tablet_bootstrap.cc:492] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7: No bootstrap required, opened a new log
I20260503 13:57:08.198217 8626 ts_tablet_manager.cc:1403] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7: Time spent bootstrapping tablet: real 0.087s user 0.025s sys 0.011s
I20260503 13:57:08.198305 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031aeee0f614639943edc1aa0067412" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:08.198937 8582 raft_consensus.cc:3060] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.201578 8626 raft_consensus.cc:359] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.202315 8626 raft_consensus.cc:385] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:08.202621 8626 raft_consensus.cc:740] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Initialized, Role: FOLLOWER
I20260503 13:57:08.203300 8626 consensus_queue.cc:260] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.205610 8626 ts_tablet_manager.cc:1434] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260503 13:57:08.206516 8626 tablet_bootstrap.cc:492] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7: Bootstrap starting.
I20260503 13:57:08.207217 8582 raft_consensus.cc:2468] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 1.
I20260503 13:57:08.208232 8475 leader_election.cc:304] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a140d8dd75a24d018e082b3ba9603de7, ff09527976954240a2b7e45977386f61; no voters: 1efe002803924b75971d7948c1675b12
I20260503 13:57:08.212185 8662 raft_consensus.cc:2804] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:08.212641 8662 raft_consensus.cc:697] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Becoming Leader. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:08.213613 8662 consensus_queue.cc:237] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.217947 8627 raft_consensus.cc:359] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.218667 8627 raft_consensus.cc:385] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:08.218969 8627 raft_consensus.cc:740] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Initialized, Role: FOLLOWER
I20260503 13:57:08.224844 8627 consensus_queue.cc:260] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.239197 8627 ts_tablet_manager.cc:1434] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12: Time spent starting tablet: real 0.065s user 0.009s sys 0.001s
I20260503 13:57:08.240305 8627 tablet_bootstrap.cc:492] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12: Bootstrap starting.
I20260503 13:57:08.239295 8626 tablet_bootstrap.cc:654] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:08.252707 8626 tablet_bootstrap.cc:492] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7: No bootstrap required, opened a new log
I20260503 13:57:08.253010 8627 tablet_bootstrap.cc:654] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:08.253162 8626 ts_tablet_manager.cc:1403] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.008s
I20260503 13:57:08.256011 8626 raft_consensus.cc:359] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.256711 8626 raft_consensus.cc:385] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:08.257033 8626 raft_consensus.cc:740] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Initialized, Role: FOLLOWER
I20260503 13:57:08.257680 8625 tablet_bootstrap.cc:492] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61: No bootstrap required, opened a new log
I20260503 13:57:08.258194 8625 ts_tablet_manager.cc:1403] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61: Time spent bootstrapping tablet: real 0.141s user 0.019s sys 0.000s
I20260503 13:57:08.257659 8626 consensus_queue.cc:260] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.260218 8626 ts_tablet_manager.cc:1434] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260503 13:57:08.261135 8626 tablet_bootstrap.cc:492] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7: Bootstrap starting.
I20260503 13:57:08.261032 8625 raft_consensus.cc:359] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.261705 8625 raft_consensus.cc:385] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:08.261966 8625 raft_consensus.cc:740] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff09527976954240a2b7e45977386f61, State: Initialized, Role: FOLLOWER
I20260503 13:57:08.262547 8625 consensus_queue.cc:260] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.266381 8627 tablet_bootstrap.cc:492] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12: No bootstrap required, opened a new log
I20260503 13:57:08.266834 8627 ts_tablet_manager.cc:1403] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s
I20260503 13:57:08.267856 8626 tablet_bootstrap.cc:654] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:08.268453 8329 catalog_manager.cc:5671] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194). New cstate: current_term: 1 leader_uuid: "a140d8dd75a24d018e082b3ba9603de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:08.269608 8627 raft_consensus.cc:359] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.270426 8627 raft_consensus.cc:385] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:08.270769 8627 raft_consensus.cc:740] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Initialized, Role: FOLLOWER
I20260503 13:57:08.266158 8330 catalog_manager.cc:5671] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff09527976954240a2b7e45977386f61 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "ff09527976954240a2b7e45977386f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: HEALTHY } } }
I20260503 13:57:08.274186 8625 ts_tablet_manager.cc:1434] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260503 13:57:08.275117 8625 tablet_bootstrap.cc:492] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61: Bootstrap starting.
I20260503 13:57:08.279282 8626 tablet_bootstrap.cc:492] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7: No bootstrap required, opened a new log
I20260503 13:57:08.279691 8626 ts_tablet_manager.cc:1403] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20260503 13:57:08.280822 8625 tablet_bootstrap.cc:654] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:08.282135 8626 raft_consensus.cc:359] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.282763 8626 raft_consensus.cc:385] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:08.283059 8626 raft_consensus.cc:740] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Initialized, Role: FOLLOWER
I20260503 13:57:08.283757 8626 consensus_queue.cc:260] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.286245 8626 ts_tablet_manager.cc:1434] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260503 13:57:08.287061 8626 tablet_bootstrap.cc:492] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7: Bootstrap starting.
I20260503 13:57:08.286695 8627 consensus_queue.cc:260] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.289069 8627 ts_tablet_manager.cc:1434] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12: Time spent starting tablet: real 0.022s user 0.007s sys 0.001s
I20260503 13:57:08.289979 8627 tablet_bootstrap.cc:492] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12: Bootstrap starting.
I20260503 13:57:08.291301 8329 catalog_manager.cc:5671] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194). New cstate: current_term: 1 leader_uuid: "a140d8dd75a24d018e082b3ba9603de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:08.292954 8626 tablet_bootstrap.cc:654] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:08.292936 8329 catalog_manager.cc:5671] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194). New cstate: current_term: 1 leader_uuid: "a140d8dd75a24d018e082b3ba9603de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:08.296625 8627 tablet_bootstrap.cc:654] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:08.303788 8626 tablet_bootstrap.cc:492] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7: No bootstrap required, opened a new log
I20260503 13:57:08.304203 8626 ts_tablet_manager.cc:1403] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260503 13:57:08.305313 8625 tablet_bootstrap.cc:492] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61: No bootstrap required, opened a new log
I20260503 13:57:08.305697 8625 ts_tablet_manager.cc:1403] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.001s
I20260503 13:57:08.306519 8626 raft_consensus.cc:359] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.307132 8626 raft_consensus.cc:385] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:08.307372 8626 raft_consensus.cc:740] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Initialized, Role: FOLLOWER
I20260503 13:57:08.308409 8626 consensus_queue.cc:260] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.310528 8627 tablet_bootstrap.cc:492] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12: No bootstrap required, opened a new log
I20260503 13:57:08.310557 8626 ts_tablet_manager.cc:1434] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260503 13:57:08.311035 8627 ts_tablet_manager.cc:1403] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.001s
I20260503 13:57:08.312646 8625 raft_consensus.cc:359] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.312923 8627 raft_consensus.cc:359] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.313289 8625 raft_consensus.cc:385] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:08.313510 8627 raft_consensus.cc:385] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:08.313560 8625 raft_consensus.cc:740] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff09527976954240a2b7e45977386f61, State: Initialized, Role: FOLLOWER
I20260503 13:57:08.313781 8627 raft_consensus.cc:740] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Initialized, Role: FOLLOWER
I20260503 13:57:08.314348 8627 consensus_queue.cc:260] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.314337 8625 consensus_queue.cc:260] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.316394 8625 ts_tablet_manager.cc:1434] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260503 13:57:08.316754 8627 ts_tablet_manager.cc:1434] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260503 13:57:08.317384 8627 tablet_bootstrap.cc:492] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12: Bootstrap starting.
I20260503 13:57:08.321005 8627 tablet_bootstrap.cc:654] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:08.331580 8627 tablet_bootstrap.cc:492] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12: No bootstrap required, opened a new log
I20260503 13:57:08.332074 8627 ts_tablet_manager.cc:1403] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.002s
I20260503 13:57:08.334522 8627 raft_consensus.cc:359] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.335148 8627 raft_consensus.cc:385] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:08.335388 8627 raft_consensus.cc:740] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Initialized, Role: FOLLOWER
I20260503 13:57:08.335944 8627 consensus_queue.cc:260] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.336702 8633 raft_consensus.cc:493] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:08.337124 8633 raft_consensus.cc:515] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.337792 8627 ts_tablet_manager.cc:1434] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260503 13:57:08.338899 8633 leader_election.cc:290] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.339634 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fe7524a546346368ea1f28ee97c221a" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7" is_pre_election: true
I20260503 13:57:08.339938 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fe7524a546346368ea1f28ee97c221a" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61" is_pre_election: true
I20260503 13:57:08.340216 8509 raft_consensus.cc:2468] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 0.
I20260503 13:57:08.340488 8582 raft_consensus.cc:2468] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 0.
I20260503 13:57:08.341463 8398 leader_election.cc:304] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7; no voters:
I20260503 13:57:08.342474 8633 raft_consensus.cc:2804] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:08.342828 8633 raft_consensus.cc:493] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:08.343276 8633 raft_consensus.cc:3060] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.347527 8633 raft_consensus.cc:515] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.349071 8633 leader_election.cc:290] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 election: Requested vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.349707 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fe7524a546346368ea1f28ee97c221a" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:08.349972 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fe7524a546346368ea1f28ee97c221a" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:08.350148 8509 raft_consensus.cc:3060] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.350533 8582 raft_consensus.cc:3060] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.355050 8582 raft_consensus.cc:2468] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 1.
I20260503 13:57:08.355113 8509 raft_consensus.cc:2468] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 1.
I20260503 13:57:08.355933 8400 leader_election.cc:304] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:08.356711 8633 raft_consensus.cc:2804] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:08.357219 8633 raft_consensus.cc:697] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Becoming Leader. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:08.358140 8633 consensus_queue.cc:237] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.360276 8661 raft_consensus.cc:493] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:08.360675 8661 raft_consensus.cc:515] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.362989 8661 leader_election.cc:290] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.363766 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76a4ada557094372a9d2a6c9b9c24cd6" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7" is_pre_election: true
I20260503 13:57:08.364001 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76a4ada557094372a9d2a6c9b9c24cd6" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61" is_pre_election: true
I20260503 13:57:08.364357 8509 raft_consensus.cc:2468] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 0.
I20260503 13:57:08.364609 8582 raft_consensus.cc:2468] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 0.
I20260503 13:57:08.365461 8398 leader_election.cc:304] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7; no voters:
I20260503 13:57:08.366398 8661 raft_consensus.cc:2804] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:08.366756 8661 raft_consensus.cc:493] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:08.367120 8661 raft_consensus.cc:3060] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.368269 8331 catalog_manager.cc:5671] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1efe002803924b75971d7948c1675b12 (127.8.26.193). New cstate: current_term: 1 leader_uuid: "1efe002803924b75971d7948c1675b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:08.373463 8661 raft_consensus.cc:515] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.375908 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76a4ada557094372a9d2a6c9b9c24cd6" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:08.376015 8661 leader_election.cc:290] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 election: Requested vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.376286 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76a4ada557094372a9d2a6c9b9c24cd6" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:08.376492 8509 raft_consensus.cc:3060] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.376838 8582 raft_consensus.cc:3060] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.382644 8582 raft_consensus.cc:2468] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 1.
I20260503 13:57:08.382644 8509 raft_consensus.cc:2468] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 1.
I20260503 13:57:08.383724 8400 leader_election.cc:304] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:08.384047 8632 raft_consensus.cc:493] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:08.384478 8661 raft_consensus.cc:2804] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:08.384929 8661 raft_consensus.cc:697] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Becoming Leader. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:08.384474 8632 raft_consensus.cc:515] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.385551 8661 consensus_queue.cc:237] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.386530 8632 leader_election.cc:290] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889)
I20260503 13:57:08.387313 8433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12" is_pre_election: true
I20260503 13:57:08.388504 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7" is_pre_election: true
I20260503 13:57:08.389285 8509 raft_consensus.cc:2393] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff09527976954240a2b7e45977386f61 in current term 1: Already voted for candidate 1efe002803924b75971d7948c1675b12 in this term.
I20260503 13:57:08.392874 8547 leader_election.cc:304] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff09527976954240a2b7e45977386f61; no voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7
I20260503 13:57:08.393884 8632 raft_consensus.cc:3060] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.395889 8331 catalog_manager.cc:5671] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1efe002803924b75971d7948c1675b12 (127.8.26.193). New cstate: current_term: 1 leader_uuid: "1efe002803924b75971d7948c1675b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:08.400148 8632 raft_consensus.cc:2749] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:57:08.432001 8662 raft_consensus.cc:493] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:08.432510 8662 raft_consensus.cc:515] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.434506 8662 leader_election.cc:290] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.435237 8433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "941dbfe411bf460e9a1563b4b946f9a7" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12" is_pre_election: true
I20260503 13:57:08.435477 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "941dbfe411bf460e9a1563b4b946f9a7" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61" is_pre_election: true
I20260503 13:57:08.435719 8433 raft_consensus.cc:2468] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 0.
I20260503 13:57:08.436105 8582 raft_consensus.cc:2468] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 0.
I20260503 13:57:08.436549 8475 leader_election.cc:304] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7; no voters:
I20260503 13:57:08.437273 8662 raft_consensus.cc:2804] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:08.437610 8662 raft_consensus.cc:493] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:08.437934 8662 raft_consensus.cc:3060] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.444196 8662 raft_consensus.cc:515] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.445982 8662 leader_election.cc:290] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.446678 8433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "941dbfe411bf460e9a1563b4b946f9a7" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:08.446907 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "941dbfe411bf460e9a1563b4b946f9a7" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:08.447111 8433 raft_consensus.cc:3060] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.447450 8582 raft_consensus.cc:3060] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.451231 8433 raft_consensus.cc:2468] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 1.
I20260503 13:57:08.452025 8475 leader_election.cc:304] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7; no voters:
I20260503 13:57:08.452611 8662 raft_consensus.cc:2804] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:08.452978 8662 raft_consensus.cc:697] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Becoming Leader. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:08.453567 8582 raft_consensus.cc:2468] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 1.
I20260503 13:57:08.453619 8662 consensus_queue.cc:237] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.455526 8661 raft_consensus.cc:493] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:08.455950 8661 raft_consensus.cc:515] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.458514 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031aeee0f614639943edc1aa0067412" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7" is_pre_election: true
I20260503 13:57:08.458806 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031aeee0f614639943edc1aa0067412" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61" is_pre_election: true
I20260503 13:57:08.459589 8582 raft_consensus.cc:2393] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1efe002803924b75971d7948c1675b12 in current term 1: Already voted for candidate a140d8dd75a24d018e082b3ba9603de7 in this term.
I20260503 13:57:08.460400 8661 leader_election.cc:290] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.461479 8400 leader_election.cc:304] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1efe002803924b75971d7948c1675b12; no voters: a140d8dd75a24d018e082b3ba9603de7, ff09527976954240a2b7e45977386f61
I20260503 13:57:08.462695 8661 raft_consensus.cc:3060] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.464169 8655 consensus_queue.cc:1048] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.468829 8661 raft_consensus.cc:2749] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:57:08.469116 8331 catalog_manager.cc:5671] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194). New cstate: current_term: 1 leader_uuid: "a140d8dd75a24d018e082b3ba9603de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:08.480794 8508 raft_consensus.cc:3060] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.482656 8633 consensus_queue.cc:1048] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.488839 8632 consensus_queue.cc:1048] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.506836 8661 consensus_queue.cc:1048] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.603754 8661 raft_consensus.cc:493] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:08.604166 8661 raft_consensus.cc:515] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.605602 8661 leader_election.cc:290] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.606520 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a140d8dd75a24d018e082b3ba9603de7" is_pre_election: true
I20260503 13:57:08.606627 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61" is_pre_election: true
I20260503 13:57:08.607401 8582 raft_consensus.cc:2393] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1efe002803924b75971d7948c1675b12 in current term 1: Already voted for candidate a140d8dd75a24d018e082b3ba9603de7 in this term.
I20260503 13:57:08.608258 8400 leader_election.cc:304] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1efe002803924b75971d7948c1675b12; no voters: a140d8dd75a24d018e082b3ba9603de7, ff09527976954240a2b7e45977386f61
I20260503 13:57:08.608982 8661 raft_consensus.cc:3060] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.613008 8661 raft_consensus.cc:2749] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:57:08.637620 8582 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a900eebd056343a89cef507a78e29e55"
dest_uuid: "ff09527976954240a2b7e45977386f61"
mode: GRACEFUL
new_leader_uuid: "1efe002803924b75971d7948c1675b12"
from {username='slave'} at 127.0.0.1:53510
I20260503 13:57:08.638197 8582 raft_consensus.cc:606] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 1 LEADER]: Received request to transfer leadership to TS 1efe002803924b75971d7948c1675b12
I20260503 13:57:08.650327 8662 consensus_queue.cc:1048] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.653749 8509 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
mode: GRACEFUL
new_leader_uuid: "1efe002803924b75971d7948c1675b12"
from {username='slave'} at 127.0.0.1:47896
I20260503 13:57:08.654328 8509 raft_consensus.cc:606] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Received request to transfer leadership to TS 1efe002803924b75971d7948c1675b12
I20260503 13:57:08.659505 8658 raft_consensus.cc:993] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7: : Instructing follower 1efe002803924b75971d7948c1675b12 to start an election
I20260503 13:57:08.659901 8658 raft_consensus.cc:1081] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Signalling peer 1efe002803924b75971d7948c1675b12 to start an election
I20260503 13:57:08.662643 8434 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888"
dest_uuid: "1efe002803924b75971d7948c1675b12"
from {username='slave'} at 127.0.0.1:36826
I20260503 13:57:08.663213 8434 raft_consensus.cc:493] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:08.663600 8434 raft_consensus.cc:3060] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.669323 8662 raft_consensus.cc:493] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:08.669757 8662 raft_consensus.cc:515] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.671423 8662 leader_election.cc:290] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.671901 8434 raft_consensus.cc:515] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.672662 8509 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9031aeee0f614639943edc1aa0067412"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
mode: GRACEFUL
new_leader_uuid: "1efe002803924b75971d7948c1675b12"
from {username='slave'} at 127.0.0.1:47896
I20260503 13:57:08.673134 8509 raft_consensus.cc:606] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Received request to transfer leadership to TS 1efe002803924b75971d7948c1675b12
I20260503 13:57:08.674855 8662 consensus_queue.cc:1048] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.675338 8631 consensus_queue.cc:1048] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.677286 8432 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566555790b4d4399bbc0571468916a89" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efe002803924b75971d7948c1675b12" is_pre_election: true
I20260503 13:57:08.677894 8432 raft_consensus.cc:2393] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a140d8dd75a24d018e082b3ba9603de7 in current term 1: Already voted for candidate ff09527976954240a2b7e45977386f61 in this term.
I20260503 13:57:08.679168 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566555790b4d4399bbc0571468916a89" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff09527976954240a2b7e45977386f61" is_pre_election: true
I20260503 13:57:08.681521 8475 leader_election.cc:304] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a140d8dd75a24d018e082b3ba9603de7; no voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61
I20260503 13:57:08.682451 8434 leader_election.cc:290] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 2 election: Requested vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.684150 8580 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "566555790b4d4399bbc0571468916a89"
dest_uuid: "ff09527976954240a2b7e45977386f61"
mode: GRACEFUL
new_leader_uuid: "a140d8dd75a24d018e082b3ba9603de7"
from {username='slave'} at 127.0.0.1:53510
I20260503 13:57:08.684821 8580 raft_consensus.cc:606] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 1 LEADER]: Received request to transfer leadership to TS a140d8dd75a24d018e082b3ba9603de7
I20260503 13:57:08.686806 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:08.687316 8509 raft_consensus.cc:3055] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:08.687611 8509 raft_consensus.cc:740] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:08.688239 8509 consensus_queue.cc:260] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.688993 8509 raft_consensus.cc:3060] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.693131 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:08.697296 8631 raft_consensus.cc:3060] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:08.703336 8433 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1fe7524a546346368ea1f28ee97c221a"
dest_uuid: "1efe002803924b75971d7948c1675b12"
mode: GRACEFUL
new_leader_uuid: "a140d8dd75a24d018e082b3ba9603de7"
from {username='slave'} at 127.0.0.1:36840
I20260503 13:57:08.703547 8631 raft_consensus.cc:2749] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:57:08.703898 8433 raft_consensus.cc:606] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Received request to transfer leadership to TS a140d8dd75a24d018e082b3ba9603de7
I20260503 13:57:08.707508 8662 consensus_queue.cc:1048] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.710569 8509 raft_consensus.cc:2468] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 2.
I20260503 13:57:08.719002 8398 leader_election.cc:304] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7; no voters: ff09527976954240a2b7e45977386f61
I20260503 13:57:08.720046 8661 raft_consensus.cc:2804] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:08.720501 8661 raft_consensus.cc:697] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Becoming Leader. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:08.721987 8661 consensus_queue.cc:237] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.726472 8434 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "76a4ada557094372a9d2a6c9b9c24cd6"
dest_uuid: "1efe002803924b75971d7948c1675b12"
mode: GRACEFUL
new_leader_uuid: "ff09527976954240a2b7e45977386f61"
from {username='slave'} at 127.0.0.1:36840
I20260503 13:57:08.727082 8434 raft_consensus.cc:606] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Received request to transfer leadership to TS ff09527976954240a2b7e45977386f61
I20260503 13:57:08.743662 8331 catalog_manager.cc:5671] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 reported cstate change: term changed from 1 to 2, leader changed from a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194) to 1efe002803924b75971d7948c1675b12 (127.8.26.193). New cstate: current_term: 2 leader_uuid: "1efe002803924b75971d7948c1675b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:08.744335 8631 consensus_queue.cc:1048] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.751511 8662 raft_consensus.cc:993] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7: : Instructing follower 1efe002803924b75971d7948c1675b12 to start an election
I20260503 13:57:08.751992 8662 raft_consensus.cc:1081] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Signalling peer 1efe002803924b75971d7948c1675b12 to start an election
I20260503 13:57:08.754065 8434 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9031aeee0f614639943edc1aa0067412"
dest_uuid: "1efe002803924b75971d7948c1675b12"
from {username='slave'} at 127.0.0.1:36826
I20260503 13:57:08.754581 8434 raft_consensus.cc:493] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:08.754921 8434 raft_consensus.cc:3060] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.760097 8434 raft_consensus.cc:515] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.761468 8433 raft_consensus.cc:1240] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Rejecting Update request from peer a140d8dd75a24d018e082b3ba9603de7 for earlier term 1. Current term is 2. Ops: []
I20260503 13:57:08.762842 8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031aeee0f614639943edc1aa0067412" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:08.763396 8582 raft_consensus.cc:3060] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.764823 8434 leader_election.cc:290] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 2 election: Requested vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.767831 8661 consensus_queue.cc:1048] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.767982 8662 consensus_queue.cc:1059] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.768921 8662 consensus_queue.cc:1048] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.769518 8582 raft_consensus.cc:2468] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 2.
I20260503 13:57:08.770153 8658 raft_consensus.cc:3055] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:08.771461 8658 raft_consensus.cc:740] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:08.772056 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031aeee0f614639943edc1aa0067412" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:08.772260 8658 consensus_queue.cc:260] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.773799 8400 leader_election.cc:304] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:08.773939 8583 raft_consensus.cc:1240] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Rejecting Update request from peer a140d8dd75a24d018e082b3ba9603de7 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260503 13:57:08.774879 8658 raft_consensus.cc:3060] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.774899 8661 raft_consensus.cc:2804] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:08.775668 8661 raft_consensus.cc:697] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Becoming Leader. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:08.776433 8661 consensus_queue.cc:237] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.777514 8688 consensus_queue.cc:1048] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.788359 8509 raft_consensus.cc:2468] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 2.
I20260503 13:57:08.792668 8330 catalog_manager.cc:5671] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 reported cstate change: term changed from 1 to 2, leader changed from a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194) to 1efe002803924b75971d7948c1675b12 (127.8.26.193). New cstate: current_term: 2 leader_uuid: "1efe002803924b75971d7948c1675b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:08.799278 8688 consensus_queue.cc:1048] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.800118 8633 consensus_queue.cc:1048] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.813611 8655 raft_consensus.cc:993] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61: : Instructing follower a140d8dd75a24d018e082b3ba9603de7 to start an election
I20260503 13:57:08.814033 8653 raft_consensus.cc:1081] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 1 LEADER]: Signalling peer a140d8dd75a24d018e082b3ba9603de7 to start an election
I20260503 13:57:08.815722 8509 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "566555790b4d4399bbc0571468916a89"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
from {username='slave'} at 127.0.0.1:47878
I20260503 13:57:08.818616 8509 raft_consensus.cc:493] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:08.819235 8509 raft_consensus.cc:3060] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.825531 8661 raft_consensus.cc:993] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12: : Instructing follower a140d8dd75a24d018e082b3ba9603de7 to start an election
I20260503 13:57:08.825973 8661 raft_consensus.cc:1081] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Signalling peer a140d8dd75a24d018e082b3ba9603de7 to start an election
I20260503 13:57:08.826088 8509 raft_consensus.cc:515] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.827438 8507 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1fe7524a546346368ea1f28ee97c221a"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
from {username='slave'} at 127.0.0.1:47890
I20260503 13:57:08.827951 8507 raft_consensus.cc:493] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:08.828346 8507 raft_consensus.cc:3060] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.829138 8509 leader_election.cc:290] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 2 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.829061 8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566555790b4d4399bbc0571468916a89" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:08.829669 8581 raft_consensus.cc:3055] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:08.829980 8581 raft_consensus.cc:740] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff09527976954240a2b7e45977386f61, State: Running, Role: LEADER
I20260503 13:57:08.830722 8581 consensus_queue.cc:260] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.831516 8581 raft_consensus.cc:3060] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.833894 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566555790b4d4399bbc0571468916a89" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:08.834425 8434 raft_consensus.cc:3060] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.836711 8581 raft_consensus.cc:2468] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 2.
I20260503 13:57:08.837672 8475 leader_election.cc:304] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a140d8dd75a24d018e082b3ba9603de7, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:08.838532 8631 raft_consensus.cc:2804] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:08.839020 8631 raft_consensus.cc:697] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 2 LEADER]: Becoming Leader. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:08.839769 8631 consensus_queue.cc:237] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.840885 8434 raft_consensus.cc:2468] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 2.
I20260503 13:57:08.841171 8507 raft_consensus.cc:515] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.846165 8507 leader_election.cc:290] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 2 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:08.847092 8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fe7524a546346368ea1f28ee97c221a" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:08.847640 8583 raft_consensus.cc:3060] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.849574 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fe7524a546346368ea1f28ee97c221a" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:08.850121 8434 raft_consensus.cc:3055] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:08.850431 8434 raft_consensus.cc:740] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:08.850634 8631 consensus_queue.cc:1048] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260503 13:57:08.851200 8434 consensus_queue.cc:260] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.852332 8434 raft_consensus.cc:3060] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.853217 8583 raft_consensus.cc:2468] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 2.
I20260503 13:57:08.854460 8475 leader_election.cc:304] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a140d8dd75a24d018e082b3ba9603de7, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:08.855365 8631 raft_consensus.cc:2804] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:08.855998 8631 raft_consensus.cc:697] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 2 LEADER]: Becoming Leader. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:08.856832 8631 consensus_queue.cc:237] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.857308 8330 catalog_manager.cc:5671] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 reported cstate change: term changed from 1 to 2, leader changed from ff09527976954240a2b7e45977386f61 (127.8.26.195) to a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194). New cstate: current_term: 2 leader_uuid: "a140d8dd75a24d018e082b3ba9603de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:08.858598 8434 raft_consensus.cc:2468] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 2.
I20260503 13:57:08.869722 8631 consensus_queue.cc:1048] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.872350 8331 catalog_manager.cc:5671] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 reported cstate change: term changed from 1 to 2, leader changed from 1efe002803924b75971d7948c1675b12 (127.8.26.193) to a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194). New cstate: current_term: 2 leader_uuid: "a140d8dd75a24d018e082b3ba9603de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:08.915684 8633 consensus_queue.cc:1048] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.923815 8705 raft_consensus.cc:993] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12: : Instructing follower ff09527976954240a2b7e45977386f61 to start an election
I20260503 13:57:08.924227 8705 raft_consensus.cc:1081] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Signalling peer ff09527976954240a2b7e45977386f61 to start an election
I20260503 13:57:08.926467 8583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "76a4ada557094372a9d2a6c9b9c24cd6"
dest_uuid: "ff09527976954240a2b7e45977386f61"
from {username='slave'} at 127.0.0.1:53496
I20260503 13:57:08.929332 8705 consensus_queue.cc:1048] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:08.930420 8583 raft_consensus.cc:493] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:08.930817 8583 raft_consensus.cc:3060] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.938313 8583 raft_consensus.cc:515] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.946252 8583 leader_election.cc:290] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 2 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889)
I20260503 13:57:08.946530 8433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76a4ada557094372a9d2a6c9b9c24cd6" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:08.947121 8507 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76a4ada557094372a9d2a6c9b9c24cd6" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:08.947121 8433 raft_consensus.cc:3055] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:08.947584 8507 raft_consensus.cc:3060] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.948020 8433 raft_consensus.cc:740] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:08.948786 8433 consensus_queue.cc:260] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.949954 8433 raft_consensus.cc:3060] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.953745 8507 raft_consensus.cc:2468] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 2.
I20260503 13:57:08.955814 8547 leader_election.cc:304] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a140d8dd75a24d018e082b3ba9603de7, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:08.956497 8433 raft_consensus.cc:2468] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 2.
I20260503 13:57:08.956633 8655 raft_consensus.cc:2804] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:08.957152 8655 raft_consensus.cc:697] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [term 2 LEADER]: Becoming Leader. State: Replica: ff09527976954240a2b7e45977386f61, State: Running, Role: LEADER
I20260503 13:57:08.957888 8655 consensus_queue.cc:237] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:08.963462 8331 catalog_manager.cc:5671] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 reported cstate change: term changed from 1 to 2, leader changed from 1efe002803924b75971d7948c1675b12 (127.8.26.193) to ff09527976954240a2b7e45977386f61 (127.8.26.195). New cstate: current_term: 2 leader_uuid: "ff09527976954240a2b7e45977386f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: HEALTHY } } }
I20260503 13:57:08.975068 8653 raft_consensus.cc:993] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61: : Instructing follower 1efe002803924b75971d7948c1675b12 to start an election
I20260503 13:57:08.975493 8655 raft_consensus.cc:1081] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 1 LEADER]: Signalling peer 1efe002803924b75971d7948c1675b12 to start an election
I20260503 13:57:08.976766 8433 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a900eebd056343a89cef507a78e29e55"
dest_uuid: "1efe002803924b75971d7948c1675b12"
from {username='slave'} at 127.0.0.1:36820
I20260503 13:57:08.977288 8433 raft_consensus.cc:493] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:08.977627 8433 raft_consensus.cc:3060] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.983913 8433 raft_consensus.cc:515] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:08.986032 8433 leader_election.cc:290] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 2 election: Requested vote from peers ff09527976954240a2b7e45977386f61 (127.8.26.195:40111), a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889)
I20260503 13:57:08.986582 8507 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a900eebd056343a89cef507a78e29e55" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:08.986347 8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a900eebd056343a89cef507a78e29e55" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:08.987133 8507 raft_consensus.cc:3060] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.987259 8583 raft_consensus.cc:3055] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:08.987603 8583 raft_consensus.cc:740] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff09527976954240a2b7e45977386f61, State: Running, Role: LEADER
I20260503 13:57:08.988297 8583 consensus_queue.cc:260] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:08.989439 8583 raft_consensus.cc:3060] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:08.993233 8507 raft_consensus.cc:2468] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 2.
I20260503 13:57:08.994243 8398 leader_election.cc:304] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7; no voters:
I20260503 13:57:08.994983 8705 raft_consensus.cc:2804] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:08.995428 8705 raft_consensus.cc:697] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Becoming Leader. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:08.995304 8583 raft_consensus.cc:2468] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 2.
I20260503 13:57:08.996222 8705 consensus_queue.cc:237] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:09.003063 8331 catalog_manager.cc:5671] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 reported cstate change: term changed from 1 to 2, leader changed from ff09527976954240a2b7e45977386f61 (127.8.26.195) to 1efe002803924b75971d7948c1675b12 (127.8.26.193). New cstate: current_term: 2 leader_uuid: "1efe002803924b75971d7948c1675b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: HEALTHY } } }
I20260503 13:57:09.234184 8583 raft_consensus.cc:1275] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Refusing update from remote peer a140d8dd75a24d018e082b3ba9603de7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:09.235285 8631 consensus_queue.cc:1048] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:09.242854 8433 raft_consensus.cc:1275] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Refusing update from remote peer a140d8dd75a24d018e082b3ba9603de7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:09.244730 8662 consensus_queue.cc:1048] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260503 13:57:09.252146 8583 raft_consensus.cc:3060] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:09.259330 8583 raft_consensus.cc:1275] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Refusing update from remote peer 1efe002803924b75971d7948c1675b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:09.260437 8705 consensus_queue.cc:1048] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:09.269173 8507 raft_consensus.cc:1275] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Refusing update from remote peer 1efe002803924b75971d7948c1675b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:09.271371 8661 consensus_queue.cc:1048] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:09.293241 8507 raft_consensus.cc:1275] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Refusing update from remote peer 1efe002803924b75971d7948c1675b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:09.294463 8661 consensus_queue.cc:1048] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:09.302894 8583 raft_consensus.cc:1275] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Refusing update from remote peer 1efe002803924b75971d7948c1675b12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:09.303957 8713 consensus_queue.cc:1048] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:09.323551 8583 raft_consensus.cc:1275] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Refusing update from remote peer a140d8dd75a24d018e082b3ba9603de7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:09.324671 8658 consensus_queue.cc:1048] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:09.331463 8433 raft_consensus.cc:1275] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Refusing update from remote peer a140d8dd75a24d018e082b3ba9603de7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:09.332712 8658 consensus_queue.cc:1048] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:09.449465 8507 raft_consensus.cc:1275] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Refusing update from remote peer ff09527976954240a2b7e45977386f61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:09.450618 8655 consensus_queue.cc:1048] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:09.457521 8433 raft_consensus.cc:1275] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Refusing update from remote peer ff09527976954240a2b7e45977386f61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:09.458925 8655 consensus_queue.cc:1048] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:09.475776 8507 raft_consensus.cc:1275] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Refusing update from remote peer 1efe002803924b75971d7948c1675b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:09.476819 8705 consensus_queue.cc:1048] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:09.483845 8583 raft_consensus.cc:1275] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Refusing update from remote peer 1efe002803924b75971d7948c1675b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:09.485298 8713 consensus_queue.cc:1048] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:11.754876 8299 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20260503 13:57:11.755553 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:11.772367 8509 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
mode: GRACEFUL
new_leader_uuid: "ff09527976954240a2b7e45977386f61"
from {username='slave'} at 127.0.0.1:38590
I20260503 13:57:11.772912 8509 raft_consensus.cc:606] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Received request to transfer leadership to TS ff09527976954240a2b7e45977386f61
I20260503 13:57:11.776221 8658 raft_consensus.cc:993] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7: : Instructing follower ff09527976954240a2b7e45977386f61 to start an election
I20260503 13:57:11.776620 8724 raft_consensus.cc:1081] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Signalling peer ff09527976954240a2b7e45977386f61 to start an election
I20260503 13:57:11.777920 8583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b"
dest_uuid: "ff09527976954240a2b7e45977386f61"
from {username='slave'} at 127.0.0.1:53488
I20260503 13:57:11.778489 8583 raft_consensus.cc:493] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:11.778841 8583 raft_consensus.cc:3060] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:11.783655 8583 raft_consensus.cc:515] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:11.785770 8583 leader_election.cc:290] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 2 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889)
I20260503 13:57:11.785773 8433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:11.786017 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:11.786468 8433 raft_consensus.cc:3060] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:11.786597 8509 raft_consensus.cc:3055] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:11.786947 8509 raft_consensus.cc:740] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:11.787631 8509 consensus_queue.cc:260] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:11.788324 8434 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a"
dest_uuid: "1efe002803924b75971d7948c1675b12"
mode: GRACEFUL
new_leader_uuid: "ff09527976954240a2b7e45977386f61"
from {username='slave'} at 127.0.0.1:43032
I20260503 13:57:11.788782 8509 raft_consensus.cc:3060] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:11.788888 8434 raft_consensus.cc:606] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Received request to transfer leadership to TS ff09527976954240a2b7e45977386f61
I20260503 13:57:11.790043 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260503 13:57:11.790586 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:11.791901 8433 raft_consensus.cc:2468] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 2.
I20260503 13:57:11.792912 8549 leader_election.cc:304] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:11.793705 8719 raft_consensus.cc:2804] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:11.794844 8509 raft_consensus.cc:2468] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 2.
I20260503 13:57:11.796011 8719 raft_consensus.cc:697] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 2 LEADER]: Becoming Leader. State: Replica: ff09527976954240a2b7e45977386f61, State: Running, Role: LEADER
I20260503 13:57:11.796734 8719 consensus_queue.cc:237] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:11.803349 8329 catalog_manager.cc:5671] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 reported cstate change: term changed from 1 to 2, leader changed from a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194) to ff09527976954240a2b7e45977386f61 (127.8.26.195). New cstate: current_term: 2 leader_uuid: "ff09527976954240a2b7e45977386f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: HEALTHY } } }
I20260503 13:57:11.809157 8720 raft_consensus.cc:993] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12: : Instructing follower ff09527976954240a2b7e45977386f61 to start an election
I20260503 13:57:11.809508 8730 raft_consensus.cc:1081] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Signalling peer ff09527976954240a2b7e45977386f61 to start an election
I20260503 13:57:11.810715 8583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a"
dest_uuid: "ff09527976954240a2b7e45977386f61"
from {username='slave'} at 127.0.0.1:53496
I20260503 13:57:11.811105 8583 raft_consensus.cc:493] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:11.811373 8583 raft_consensus.cc:3060] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:11.815893 8583 raft_consensus.cc:515] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:11.817198 8583 leader_election.cc:290] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 2 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889)
I20260503 13:57:11.818009 8433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:11.818073 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:11.818643 8433 raft_consensus.cc:3055] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:11.818691 8509 raft_consensus.cc:3060] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:11.818939 8433 raft_consensus.cc:740] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:11.819546 8433 consensus_queue.cc:260] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:11.820325 8433 raft_consensus.cc:3060] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:11.822906 8509 raft_consensus.cc:2468] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 2.
I20260503 13:57:11.823691 8547 leader_election.cc:304] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a140d8dd75a24d018e082b3ba9603de7, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:11.824196 8433 raft_consensus.cc:2468] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 2.
I20260503 13:57:11.824272 8719 raft_consensus.cc:2804] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:11.824843 8719 raft_consensus.cc:697] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 2 LEADER]: Becoming Leader. State: Replica: ff09527976954240a2b7e45977386f61, State: Running, Role: LEADER
I20260503 13:57:11.825618 8719 consensus_queue.cc:237] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:11.831748 8329 catalog_manager.cc:5671] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 reported cstate change: term changed from 1 to 2, leader changed from 1efe002803924b75971d7948c1675b12 (127.8.26.193) to ff09527976954240a2b7e45977386f61 (127.8.26.195). New cstate: current_term: 2 leader_uuid: "ff09527976954240a2b7e45977386f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: HEALTHY } } }
I20260503 13:57:12.221472 8509 raft_consensus.cc:1275] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Refusing update from remote peer ff09527976954240a2b7e45977386f61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:12.222523 8719 consensus_queue.cc:1048] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:12.232262 8433 raft_consensus.cc:1275] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Refusing update from remote peer ff09527976954240a2b7e45977386f61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:12.233507 8726 consensus_queue.cc:1048] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:12.279518 8433 raft_consensus.cc:1275] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Refusing update from remote peer ff09527976954240a2b7e45977386f61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:12.280601 8726 consensus_queue.cc:1048] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:12.288885 8509 raft_consensus.cc:1275] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Refusing update from remote peer ff09527976954240a2b7e45977386f61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:12.290261 8740 consensus_queue.cc:1048] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:12.791435 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:12.797688 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:12.798039 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:13.798913 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:13.804234 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:13.804625 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:14.805373 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:14.810173 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:14.810521 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:15.811466 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:15.816220 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:15.816551 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:16.817471 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:16.822237 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:16.822582 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:17.823343 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:17.827986 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:17.828303 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:18.829155 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:18.833863 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:18.834180 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:19.834877 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:19.839422 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:19.839721 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:20.840404 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:20.844914 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:20.845212 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:21.868312 8299 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20260503 13:57:21.872043 8434 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a900eebd056343a89cef507a78e29e55"
dest_uuid: "1efe002803924b75971d7948c1675b12"
mode: GRACEFUL
new_leader_uuid: "a140d8dd75a24d018e082b3ba9603de7"
from {username='slave'} at 127.0.0.1:36840
I20260503 13:57:21.872676 8434 raft_consensus.cc:606] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Received request to transfer leadership to TS a140d8dd75a24d018e082b3ba9603de7
I20260503 13:57:21.876498 8583 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a"
dest_uuid: "ff09527976954240a2b7e45977386f61"
mode: GRACEFUL
new_leader_uuid: "a140d8dd75a24d018e082b3ba9603de7"
from {username='slave'} at 127.0.0.1:53510
I20260503 13:57:21.876910 8583 raft_consensus.cc:606] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 2 LEADER]: Received request to transfer leadership to TS a140d8dd75a24d018e082b3ba9603de7
I20260503 13:57:21.880527 8434 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888"
dest_uuid: "1efe002803924b75971d7948c1675b12"
mode: GRACEFUL
new_leader_uuid: "a140d8dd75a24d018e082b3ba9603de7"
from {username='slave'} at 127.0.0.1:36840
I20260503 13:57:21.880934 8434 raft_consensus.cc:606] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Received request to transfer leadership to TS a140d8dd75a24d018e082b3ba9603de7
I20260503 13:57:21.884637 8434 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9031aeee0f614639943edc1aa0067412"
dest_uuid: "1efe002803924b75971d7948c1675b12"
mode: GRACEFUL
new_leader_uuid: "a140d8dd75a24d018e082b3ba9603de7"
from {username='slave'} at 127.0.0.1:36840
I20260503 13:57:21.884994 8434 raft_consensus.cc:606] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Received request to transfer leadership to TS a140d8dd75a24d018e082b3ba9603de7
I20260503 13:57:21.894790 8583 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b"
dest_uuid: "ff09527976954240a2b7e45977386f61"
mode: GRACEFUL
new_leader_uuid: "a140d8dd75a24d018e082b3ba9603de7"
from {username='slave'} at 127.0.0.1:53510
I20260503 13:57:21.895139 8583 raft_consensus.cc:606] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 2 LEADER]: Received request to transfer leadership to TS a140d8dd75a24d018e082b3ba9603de7
I20260503 13:57:21.946125 8719 raft_consensus.cc:993] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61: : Instructing follower a140d8dd75a24d018e082b3ba9603de7 to start an election
I20260503 13:57:21.946688 8749 raft_consensus.cc:1081] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 2 LEADER]: Signalling peer a140d8dd75a24d018e082b3ba9603de7 to start an election
I20260503 13:57:21.948179 8509 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
from {username='slave'} at 127.0.0.1:47878
I20260503 13:57:21.948630 8509 raft_consensus.cc:493] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:21.948933 8509 raft_consensus.cc:3060] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:21.953356 8509 raft_consensus.cc:515] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:21.954756 8509 leader_election.cc:290] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 3 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:21.955438 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:21.955605 8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:21.956054 8434 raft_consensus.cc:3060] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:21.956043 8583 raft_consensus.cc:3055] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:57:21.956571 8583 raft_consensus.cc:740] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ff09527976954240a2b7e45977386f61, State: Running, Role: LEADER
I20260503 13:57:21.957144 8583 consensus_queue.cc:260] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:21.958020 8583 raft_consensus.cc:3060] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:21.960469 8434 raft_consensus.cc:2468] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 3.
I20260503 13:57:21.961555 8475 leader_election.cc:304] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7; no voters:
I20260503 13:57:21.962025 8583 raft_consensus.cc:2468] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 3.
I20260503 13:57:21.962320 8742 raft_consensus.cc:2804] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:57:21.963217 8742 raft_consensus.cc:697] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Becoming Leader. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:21.964020 8742 consensus_queue.cc:237] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:21.970537 8329 catalog_manager.cc:5671] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 reported cstate change: term changed from 2 to 3, leader changed from ff09527976954240a2b7e45977386f61 (127.8.26.195) to a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194). New cstate: current_term: 3 leader_uuid: "a140d8dd75a24d018e082b3ba9603de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:22.069300 8719 raft_consensus.cc:993] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61: : Instructing follower a140d8dd75a24d018e082b3ba9603de7 to start an election
I20260503 13:57:22.069690 8749 raft_consensus.cc:1081] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 2 LEADER]: Signalling peer a140d8dd75a24d018e082b3ba9603de7 to start an election
I20260503 13:57:22.070924 8509 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
from {username='slave'} at 127.0.0.1:47878
I20260503 13:57:22.071329 8509 raft_consensus.cc:493] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:22.071569 8509 raft_consensus.cc:3060] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:22.075280 8509 raft_consensus.cc:515] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:22.076505 8509 leader_election.cc:290] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 3 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:22.077116 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:22.077395 8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:22.077594 8434 raft_consensus.cc:3060] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:22.077819 8583 raft_consensus.cc:3055] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:57:22.078048 8583 raft_consensus.cc:740] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ff09527976954240a2b7e45977386f61, State: Running, Role: LEADER
I20260503 13:57:22.078644 8583 consensus_queue.cc:260] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:22.079507 8583 raft_consensus.cc:3060] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:22.082334 8756 raft_consensus.cc:993] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12: : Instructing follower a140d8dd75a24d018e082b3ba9603de7 to start an election
I20260503 13:57:22.082952 8743 raft_consensus.cc:1081] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Signalling peer a140d8dd75a24d018e082b3ba9603de7 to start an election
I20260503 13:57:22.082984 8434 raft_consensus.cc:2468] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 3.
I20260503 13:57:22.083916 8475 leader_election.cc:304] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, a140d8dd75a24d018e082b3ba9603de7; no voters:
I20260503 13:57:22.084606 8742 raft_consensus.cc:2804] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:57:22.084650 8509 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9031aeee0f614639943edc1aa0067412"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
from {username='slave'} at 127.0.0.1:47890
I20260503 13:57:22.085032 8742 raft_consensus.cc:697] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Becoming Leader. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:22.085203 8509 raft_consensus.cc:493] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:22.085230 8583 raft_consensus.cc:2468] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 3.
I20260503 13:57:22.085672 8509 raft_consensus.cc:3060] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:22.085847 8742 consensus_queue.cc:237] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:22.089663 8509 raft_consensus.cc:515] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:22.091166 8509 leader_election.cc:290] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 3 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:22.091958 8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031aeee0f614639943edc1aa0067412" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:22.092288 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031aeee0f614639943edc1aa0067412" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:22.092471 8583 raft_consensus.cc:3060] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:22.092813 8434 raft_consensus.cc:3055] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:57:22.093165 8434 raft_consensus.cc:740] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:22.092763 8329 catalog_manager.cc:5671] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 reported cstate change: term changed from 2 to 3, leader changed from ff09527976954240a2b7e45977386f61 (127.8.26.195) to a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194). New cstate: current_term: 3 leader_uuid: "a140d8dd75a24d018e082b3ba9603de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:22.093853 8434 consensus_queue.cc:260] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:22.095028 8434 raft_consensus.cc:3060] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:22.097682 8583 raft_consensus.cc:2468] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 3.
I20260503 13:57:22.098681 8475 leader_election.cc:304] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a140d8dd75a24d018e082b3ba9603de7, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:22.099318 8742 raft_consensus.cc:2804] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:57:22.099704 8742 raft_consensus.cc:697] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Becoming Leader. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:22.100376 8742 consensus_queue.cc:237] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:22.104337 8434 raft_consensus.cc:2468] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 3.
I20260503 13:57:22.106336 8329 catalog_manager.cc:5671] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 reported cstate change: term changed from 2 to 3, leader changed from 1efe002803924b75971d7948c1675b12 (127.8.26.193) to a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194). New cstate: current_term: 3 leader_uuid: "a140d8dd75a24d018e082b3ba9603de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:22.191349 8756 raft_consensus.cc:993] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12: : Instructing follower a140d8dd75a24d018e082b3ba9603de7 to start an election
I20260503 13:57:22.191838 8743 raft_consensus.cc:1081] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Signalling peer a140d8dd75a24d018e082b3ba9603de7 to start an election
I20260503 13:57:22.193245 8509 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a900eebd056343a89cef507a78e29e55"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
from {username='slave'} at 127.0.0.1:47890
I20260503 13:57:22.193753 8509 raft_consensus.cc:493] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:22.193994 8509 raft_consensus.cc:3060] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:22.197786 8509 raft_consensus.cc:515] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:22.199038 8509 leader_election.cc:290] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 3 election: Requested vote from peers ff09527976954240a2b7e45977386f61 (127.8.26.195:40111), 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633)
I20260503 13:57:22.199910 8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a900eebd056343a89cef507a78e29e55" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:22.200004 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a900eebd056343a89cef507a78e29e55" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:22.200453 8583 raft_consensus.cc:3060] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:22.200520 8434 raft_consensus.cc:3055] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:57:22.200897 8434 raft_consensus.cc:740] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:22.201653 8434 consensus_queue.cc:260] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:22.202684 8434 raft_consensus.cc:3060] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:22.204596 8583 raft_consensus.cc:2468] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 3.
I20260503 13:57:22.205323 8475 leader_election.cc:304] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a140d8dd75a24d018e082b3ba9603de7, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:22.205847 8742 raft_consensus.cc:2804] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:57:22.206173 8742 raft_consensus.cc:697] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Becoming Leader. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:22.206833 8742 consensus_queue.cc:237] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:22.207099 8434 raft_consensus.cc:2468] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 3.
I20260503 13:57:22.212707 8329 catalog_manager.cc:5671] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 reported cstate change: term changed from 2 to 3, leader changed from 1efe002803924b75971d7948c1675b12 (127.8.26.193) to a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194). New cstate: current_term: 3 leader_uuid: "a140d8dd75a24d018e082b3ba9603de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:22.298003 8756 raft_consensus.cc:993] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12: : Instructing follower a140d8dd75a24d018e082b3ba9603de7 to start an election
I20260503 13:57:22.298442 8743 raft_consensus.cc:1081] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Signalling peer a140d8dd75a24d018e082b3ba9603de7 to start an election
I20260503 13:57:22.299654 8509 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
from {username='slave'} at 127.0.0.1:47890
I20260503 13:57:22.300050 8509 raft_consensus.cc:493] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:22.300320 8509 raft_consensus.cc:3060] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:22.303963 8509 raft_consensus.cc:515] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:22.305141 8509 leader_election.cc:290] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 3 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:22.305866 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:22.306003 8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd326985c85427392b9ecb990c3d888" candidate_uuid: "a140d8dd75a24d018e082b3ba9603de7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:22.306284 8434 raft_consensus.cc:3055] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:57:22.306429 8583 raft_consensus.cc:3060] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:22.306542 8434 raft_consensus.cc:740] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:22.307139 8434 consensus_queue.cc:260] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:22.307885 8434 raft_consensus.cc:3060] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:22.310088 8583 raft_consensus.cc:2468] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 3.
I20260503 13:57:22.310829 8475 leader_election.cc:304] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a140d8dd75a24d018e082b3ba9603de7, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:22.311487 8742 raft_consensus.cc:2804] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:57:22.311736 8434 raft_consensus.cc:2468] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a140d8dd75a24d018e082b3ba9603de7 in term 3.
I20260503 13:57:22.311818 8742 raft_consensus.cc:697] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Becoming Leader. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:22.312608 8742 consensus_queue.cc:237] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:22.318425 8329 catalog_manager.cc:5671] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 reported cstate change: term changed from 2 to 3, leader changed from 1efe002803924b75971d7948c1675b12 (127.8.26.193) to a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194). New cstate: current_term: 3 leader_uuid: "a140d8dd75a24d018e082b3ba9603de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:22.524901 8583 raft_consensus.cc:1275] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Refusing update from remote peer a140d8dd75a24d018e082b3ba9603de7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:22.525974 8750 consensus_queue.cc:1048] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:22.534871 8434 raft_consensus.cc:1275] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Refusing update from remote peer a140d8dd75a24d018e082b3ba9603de7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:22.538393 8742 consensus_queue.cc:1048] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:22.545249 8434 raft_consensus.cc:1275] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Refusing update from remote peer a140d8dd75a24d018e082b3ba9603de7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:22.546237 8742 consensus_queue.cc:1048] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:22.553166 8583 raft_consensus.cc:1275] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Refusing update from remote peer a140d8dd75a24d018e082b3ba9603de7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:22.554644 8750 consensus_queue.cc:1048] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:22.589215 8434 raft_consensus.cc:1275] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Refusing update from remote peer a140d8dd75a24d018e082b3ba9603de7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:22.590185 8750 consensus_queue.cc:1048] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:22.597857 8583 raft_consensus.cc:1275] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Refusing update from remote peer a140d8dd75a24d018e082b3ba9603de7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:22.598879 8750 consensus_queue.cc:1048] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:22.704298 8583 raft_consensus.cc:1275] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Refusing update from remote peer a140d8dd75a24d018e082b3ba9603de7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:22.705497 8750 consensus_queue.cc:1048] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:22.714900 8434 raft_consensus.cc:1275] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Refusing update from remote peer a140d8dd75a24d018e082b3ba9603de7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:22.715857 8764 consensus_queue.cc:1048] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:22.765465 8583 raft_consensus.cc:1275] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Refusing update from remote peer a140d8dd75a24d018e082b3ba9603de7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:22.766484 8764 consensus_queue.cc:1048] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:22.773844 8434 raft_consensus.cc:1275] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Refusing update from remote peer a140d8dd75a24d018e082b3ba9603de7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:22.775125 8764 consensus_queue.cc:1048] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:24.920250 8299 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20260503 13:57:24.920888 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:24.926653 8509 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1fe7524a546346368ea1f28ee97c221a"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
mode: GRACEFUL
new_leader_uuid: "1efe002803924b75971d7948c1675b12"
from {username='slave'} at 127.0.0.1:38590
I20260503 13:57:24.927049 8509 raft_consensus.cc:606] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 2 LEADER]: Received request to transfer leadership to TS 1efe002803924b75971d7948c1675b12
I20260503 13:57:24.928861 8509 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "566555790b4d4399bbc0571468916a89"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
mode: GRACEFUL
new_leader_uuid: "ff09527976954240a2b7e45977386f61"
from {username='slave'} at 127.0.0.1:38590
I20260503 13:57:24.929216 8509 raft_consensus.cc:606] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 2 LEADER]: Received request to transfer leadership to TS ff09527976954240a2b7e45977386f61
I20260503 13:57:24.930787 8509 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a900eebd056343a89cef507a78e29e55"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
mode: GRACEFUL
new_leader_uuid: "ff09527976954240a2b7e45977386f61"
from {username='slave'} at 127.0.0.1:38590
I20260503 13:57:24.931126 8509 raft_consensus.cc:606] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Received request to transfer leadership to TS ff09527976954240a2b7e45977386f61
I20260503 13:57:24.932848 8509 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
mode: GRACEFUL
new_leader_uuid: "1efe002803924b75971d7948c1675b12"
from {username='slave'} at 127.0.0.1:38590
I20260503 13:57:24.933172 8509 raft_consensus.cc:606] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Received request to transfer leadership to TS 1efe002803924b75971d7948c1675b12
I20260503 13:57:24.934945 8509 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a"
dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
mode: GRACEFUL
new_leader_uuid: "1efe002803924b75971d7948c1675b12"
from {username='slave'} at 127.0.0.1:38590
I20260503 13:57:24.935276 8509 raft_consensus.cc:606] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Received request to transfer leadership to TS 1efe002803924b75971d7948c1675b12
I20260503 13:57:24.935997 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260503 13:57:24.936408 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:24.942800 8750 raft_consensus.cc:993] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7: : Instructing follower ff09527976954240a2b7e45977386f61 to start an election
I20260503 13:57:24.943179 8776 raft_consensus.cc:1081] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 2 LEADER]: Signalling peer ff09527976954240a2b7e45977386f61 to start an election
I20260503 13:57:24.944278 8583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "566555790b4d4399bbc0571468916a89"
dest_uuid: "ff09527976954240a2b7e45977386f61"
from {username='slave'} at 127.0.0.1:53488
I20260503 13:57:24.944692 8583 raft_consensus.cc:493] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:24.944917 8583 raft_consensus.cc:3060] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:24.948446 8583 raft_consensus.cc:515] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:24.950384 8583 leader_election.cc:290] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 3 election: Requested vote from peers 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633), a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889)
I20260503 13:57:24.951257 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566555790b4d4399bbc0571468916a89" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:24.951457 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566555790b4d4399bbc0571468916a89" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:24.951747 8434 raft_consensus.cc:3060] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:24.951884 8509 raft_consensus.cc:3055] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:57:24.952199 8509 raft_consensus.cc:740] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:24.952849 8509 consensus_queue.cc:260] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:24.953974 8509 raft_consensus.cc:3060] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:24.957309 8434 raft_consensus.cc:2468] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 3.
I20260503 13:57:24.958379 8549 leader_election.cc:304] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:24.959117 8749 raft_consensus.cc:2804] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:57:24.959414 8509 raft_consensus.cc:2468] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 3.
I20260503 13:57:24.960039 8749 raft_consensus.cc:697] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 3 LEADER]: Becoming Leader. State: Replica: ff09527976954240a2b7e45977386f61, State: Running, Role: LEADER
I20260503 13:57:24.960855 8749 consensus_queue.cc:237] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:24.967017 8331 catalog_manager.cc:5671] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 reported cstate change: term changed from 2 to 3, leader changed from a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194) to ff09527976954240a2b7e45977386f61 (127.8.26.195). New cstate: current_term: 3 leader_uuid: "ff09527976954240a2b7e45977386f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: HEALTHY } } }
I20260503 13:57:25.135219 8776 raft_consensus.cc:993] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7: : Instructing follower 1efe002803924b75971d7948c1675b12 to start an election
I20260503 13:57:25.135636 8750 raft_consensus.cc:1081] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Signalling peer 1efe002803924b75971d7948c1675b12 to start an election
I20260503 13:57:25.136790 8434 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a"
dest_uuid: "1efe002803924b75971d7948c1675b12"
from {username='slave'} at 127.0.0.1:36826
I20260503 13:57:25.137161 8434 raft_consensus.cc:493] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:25.137363 8434 raft_consensus.cc:3060] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:25.140908 8434 raft_consensus.cc:515] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:25.142498 8434 leader_election.cc:290] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 4 election: Requested vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:25.143142 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:25.143443 8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2a6dc6f02264540ac18b8a431c42e4a" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:25.143687 8509 raft_consensus.cc:3055] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:57:25.143926 8583 raft_consensus.cc:3060] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:25.143970 8509 raft_consensus.cc:740] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:25.144711 8509 consensus_queue.cc:260] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:25.145750 8509 raft_consensus.cc:3060] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:25.148842 8776 raft_consensus.cc:993] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7: : Instructing follower 1efe002803924b75971d7948c1675b12 to start an election
I20260503 13:57:25.149202 8750 raft_consensus.cc:1081] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Signalling peer 1efe002803924b75971d7948c1675b12 to start an election
I20260503 13:57:25.149410 8583 raft_consensus.cc:2468] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 4.
I20260503 13:57:25.150527 8400 leader_election.cc:304] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:25.151358 8509 raft_consensus.cc:2468] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 4.
I20260503 13:57:25.152072 8783 raft_consensus.cc:2804] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:57:25.152024 8434 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b"
dest_uuid: "1efe002803924b75971d7948c1675b12"
from {username='slave'} at 127.0.0.1:36826
I20260503 13:57:25.152639 8434 raft_consensus.cc:493] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:25.152874 8434 raft_consensus.cc:3060] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:25.153256 8783 raft_consensus.cc:697] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 4 LEADER]: Becoming Leader. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:25.154011 8783 consensus_queue.cc:237] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:25.156481 8434 raft_consensus.cc:515] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:25.158113 8434 leader_election.cc:290] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 4 election: Requested vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:25.158674 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:25.159081 8509 raft_consensus.cc:3055] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:57:25.159202 8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7caf0031e354b8ca998e7640505cb9b" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:25.159373 8509 raft_consensus.cc:740] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:25.159690 8583 raft_consensus.cc:3060] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:25.160010 8509 consensus_queue.cc:260] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:25.160358 8331 catalog_manager.cc:5671] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 reported cstate change: term changed from 3 to 4, leader changed from a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194) to 1efe002803924b75971d7948c1675b12 (127.8.26.193). New cstate: current_term: 4 leader_uuid: "1efe002803924b75971d7948c1675b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:25.161057 8509 raft_consensus.cc:3060] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:25.165174 8583 raft_consensus.cc:2468] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 4.
I20260503 13:57:25.166164 8400 leader_election.cc:304] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:25.166576 8509 raft_consensus.cc:2468] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 4.
I20260503 13:57:25.166877 8783 raft_consensus.cc:2804] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:57:25.167366 8783 raft_consensus.cc:697] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 4 LEADER]: Becoming Leader. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:25.168071 8783 consensus_queue.cc:237] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:25.173934 8331 catalog_manager.cc:5671] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 reported cstate change: term changed from 3 to 4, leader changed from a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194) to 1efe002803924b75971d7948c1675b12 (127.8.26.193). New cstate: current_term: 4 leader_uuid: "1efe002803924b75971d7948c1675b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:25.255236 8776 raft_consensus.cc:993] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7: : Instructing follower 1efe002803924b75971d7948c1675b12 to start an election
I20260503 13:57:25.255841 8750 raft_consensus.cc:1081] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 2 LEADER]: Signalling peer 1efe002803924b75971d7948c1675b12 to start an election
I20260503 13:57:25.257145 8434 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1fe7524a546346368ea1f28ee97c221a"
dest_uuid: "1efe002803924b75971d7948c1675b12"
from {username='slave'} at 127.0.0.1:36826
I20260503 13:57:25.257658 8434 raft_consensus.cc:493] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:25.257938 8434 raft_consensus.cc:3060] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:25.262878 8434 raft_consensus.cc:515] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:25.264442 8434 leader_election.cc:290] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 3 election: Requested vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), ff09527976954240a2b7e45977386f61 (127.8.26.195:40111)
I20260503 13:57:25.266371 8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fe7524a546346368ea1f28ee97c221a" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ff09527976954240a2b7e45977386f61"
I20260503 13:57:25.266115 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fe7524a546346368ea1f28ee97c221a" candidate_uuid: "1efe002803924b75971d7948c1675b12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:25.267025 8583 raft_consensus.cc:3060] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:25.267007 8509 raft_consensus.cc:3055] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:57:25.267531 8509 raft_consensus.cc:740] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:25.268285 8509 consensus_queue.cc:260] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:25.269667 8509 raft_consensus.cc:3060] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:25.273702 8583 raft_consensus.cc:2468] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 3.
I20260503 13:57:25.274998 8400 leader_election.cc:304] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:25.276229 8509 raft_consensus.cc:2468] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe002803924b75971d7948c1675b12 in term 3.
I20260503 13:57:25.276882 8783 raft_consensus.cc:2804] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:57:25.277858 8783 raft_consensus.cc:697] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 3 LEADER]: Becoming Leader. State: Replica: 1efe002803924b75971d7948c1675b12, State: Running, Role: LEADER
I20260503 13:57:25.278920 8783 consensus_queue.cc:237] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } }
I20260503 13:57:25.287083 8331 catalog_manager.cc:5671] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 reported cstate change: term changed from 2 to 3, leader changed from a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194) to 1efe002803924b75971d7948c1675b12 (127.8.26.193). New cstate: current_term: 3 leader_uuid: "1efe002803924b75971d7948c1675b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:25.303936 8776 raft_consensus.cc:993] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7: : Instructing follower ff09527976954240a2b7e45977386f61 to start an election
I20260503 13:57:25.304313 8750 raft_consensus.cc:1081] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Signalling peer ff09527976954240a2b7e45977386f61 to start an election
I20260503 13:57:25.305485 8583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a900eebd056343a89cef507a78e29e55"
dest_uuid: "ff09527976954240a2b7e45977386f61"
from {username='slave'} at 127.0.0.1:53488
I20260503 13:57:25.305828 8583 raft_consensus.cc:493] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:25.306030 8583 raft_consensus.cc:3060] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:25.309741 8583 raft_consensus.cc:515] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:25.311008 8583 leader_election.cc:290] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 4 election: Requested vote from peers a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194:36889), 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633)
I20260503 13:57:25.311900 8509 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a900eebd056343a89cef507a78e29e55" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a140d8dd75a24d018e082b3ba9603de7"
I20260503 13:57:25.312021 8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a900eebd056343a89cef507a78e29e55" candidate_uuid: "ff09527976954240a2b7e45977386f61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1efe002803924b75971d7948c1675b12"
I20260503 13:57:25.312280 8509 raft_consensus.cc:3055] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:57:25.312493 8434 raft_consensus.cc:3060] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:25.312541 8509 raft_consensus.cc:740] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a140d8dd75a24d018e082b3ba9603de7, State: Running, Role: LEADER
I20260503 13:57:25.313254 8509 consensus_queue.cc:260] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:25.314142 8509 raft_consensus.cc:3060] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:25.316634 8434 raft_consensus.cc:2468] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 4.
I20260503 13:57:25.317468 8549 leader_election.cc:304] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe002803924b75971d7948c1675b12, ff09527976954240a2b7e45977386f61; no voters:
I20260503 13:57:25.318055 8509 raft_consensus.cc:2468] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff09527976954240a2b7e45977386f61 in term 4.
I20260503 13:57:25.318181 8749 raft_consensus.cc:2804] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:57:25.318653 8749 raft_consensus.cc:697] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 4 LEADER]: Becoming Leader. State: Replica: ff09527976954240a2b7e45977386f61, State: Running, Role: LEADER
I20260503 13:57:25.319586 8749 consensus_queue.cc:237] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } }
I20260503 13:57:25.325996 8331 catalog_manager.cc:5671] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 reported cstate change: term changed from 3 to 4, leader changed from a140d8dd75a24d018e082b3ba9603de7 (127.8.26.194) to ff09527976954240a2b7e45977386f61 (127.8.26.195). New cstate: current_term: 4 leader_uuid: "ff09527976954240a2b7e45977386f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:25.393177 8509 raft_consensus.cc:1275] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Refusing update from remote peer ff09527976954240a2b7e45977386f61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:25.394124 8749 consensus_queue.cc:1048] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:25.402958 8434 raft_consensus.cc:1275] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Refusing update from remote peer ff09527976954240a2b7e45977386f61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:25.405377 8777 consensus_queue.cc:1048] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260503 13:57:25.728148 8507 raft_consensus.cc:1275] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 4 FOLLOWER]: Refusing update from remote peer 1efe002803924b75971d7948c1675b12: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:57:25.729604 8783 consensus_queue.cc:1048] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:57:25.730320 8582 raft_consensus.cc:1275] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 4 FOLLOWER]: Refusing update from remote peer 1efe002803924b75971d7948c1675b12: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:57:25.731930 8784 consensus_queue.cc:1048] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:57:25.733103 8583 raft_consensus.cc:1275] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 4 FOLLOWER]: Refusing update from remote peer 1efe002803924b75971d7948c1675b12: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:57:25.736332 8784 consensus_queue.cc:1048] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:57:25.738036 8581 raft_consensus.cc:1275] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Refusing update from remote peer 1efe002803924b75971d7948c1675b12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:25.741698 8509 raft_consensus.cc:1275] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 4 FOLLOWER]: Refusing update from remote peer 1efe002803924b75971d7948c1675b12: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:57:25.742130 8784 consensus_queue.cc:1048] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff09527976954240a2b7e45977386f61" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:25.746301 8783 consensus_queue.cc:1048] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:57:25.755865 8508 raft_consensus.cc:1275] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Refusing update from remote peer 1efe002803924b75971d7948c1675b12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:25.757543 8796 consensus_queue.cc:1048] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:25.850476 8508 raft_consensus.cc:1275] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 4 FOLLOWER]: Refusing update from remote peer ff09527976954240a2b7e45977386f61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:57:25.851464 8777 consensus_queue.cc:1048] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a140d8dd75a24d018e082b3ba9603de7" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:57:25.860064 8434 raft_consensus.cc:1275] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 4 FOLLOWER]: Refusing update from remote peer ff09527976954240a2b7e45977386f61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:57:25.862824 8809 consensus_queue.cc:1048] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efe002803924b75971d7948c1675b12" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260503 13:57:26.022022 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:26.026708 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:26.027035 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260503 13:57:26.663879 8455 debug-util.cc:398] Leaking SignalData structure 0x7b08000e6060 after lost signal to thread 8302
W20260503 13:57:26.664546 8455 debug-util.cc:398] Leaking SignalData structure 0x7b08000e6740 after lost signal to thread 8365
W20260503 13:57:26.665277 8455 debug-util.cc:398] Leaking SignalData structure 0x7b08001d4840 after lost signal to thread 8458
W20260503 13:57:26.665933 8455 debug-util.cc:398] Leaking SignalData structure 0x7b08001d4a00 after lost signal to thread 8533
W20260503 13:57:26.666551 8455 debug-util.cc:398] Leaking SignalData structure 0x7b08001900a0 after lost signal to thread 8607
I20260503 13:57:27.028012 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:27.033139 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:27.033492 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:28.034343 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:28.039033 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:28.039379 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:29.040174 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:29.044771 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:29.045166 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:30.046020 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:30.050576 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:30.050927 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:31.051817 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:31.056366 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:31.056741 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:32.057617 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:32.062320 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:32.062705 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:33.063694 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:33.068596 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:33.068954 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:34.069828 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:34.074347 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:34.074707 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:35.098430 8299 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20260503 13:57:35.098913 8299 tablet_server.cc:179] TabletServer@127.8.26.193:0 shutting down...
I20260503 13:57:35.118798 8299 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260503 13:57:35.119546 8299 tablet_replica.cc:333] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12: stopping tablet replica
I20260503 13:57:35.120193 8299 raft_consensus.cc:2243] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.120729 8299 raft_consensus.cc:2272] T 76a4ada557094372a9d2a6c9b9c24cd6 P 1efe002803924b75971d7948c1675b12 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.122864 8299 tablet_replica.cc:333] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12: stopping tablet replica
I20260503 13:57:35.123317 8299 raft_consensus.cc:2243] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.123642 8299 raft_consensus.cc:2272] T 941dbfe411bf460e9a1563b4b946f9a7 P 1efe002803924b75971d7948c1675b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.125182 8299 tablet_replica.cc:333] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12: stopping tablet replica
I20260503 13:57:35.125605 8299 raft_consensus.cc:2243] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.125948 8299 raft_consensus.cc:2272] T 9031aeee0f614639943edc1aa0067412 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.127404 8299 tablet_replica.cc:333] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12: stopping tablet replica
I20260503 13:57:35.127835 8299 raft_consensus.cc:2243] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 4 LEADER]: Raft consensus shutting down.
I20260503 13:57:35.128424 8299 raft_consensus.cc:2272] T d7caf0031e354b8ca998e7640505cb9b P 1efe002803924b75971d7948c1675b12 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.129911 8299 tablet_replica.cc:333] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12: stopping tablet replica
I20260503 13:57:35.130342 8299 raft_consensus.cc:2243] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 4 LEADER]: Raft consensus shutting down.
I20260503 13:57:35.130946 8299 raft_consensus.cc:2272] T f2a6dc6f02264540ac18b8a431c42e4a P 1efe002803924b75971d7948c1675b12 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.132290 8299 tablet_replica.cc:333] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12: stopping tablet replica
I20260503 13:57:35.132791 8299 raft_consensus.cc:2243] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.133198 8299 raft_consensus.cc:2272] T 566555790b4d4399bbc0571468916a89 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.134452 8299 tablet_replica.cc:333] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12: stopping tablet replica
I20260503 13:57:35.134860 8299 raft_consensus.cc:2243] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 3 LEADER]: Raft consensus shutting down.
I20260503 13:57:35.135418 8299 raft_consensus.cc:2272] T 1fe7524a546346368ea1f28ee97c221a P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.137116 8299 tablet_replica.cc:333] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12: stopping tablet replica
I20260503 13:57:35.137491 8299 raft_consensus.cc:2243] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.137884 8299 raft_consensus.cc:2272] T cbd326985c85427392b9ecb990c3d888 P 1efe002803924b75971d7948c1675b12 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.139282 8299 tablet_replica.cc:333] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12: stopping tablet replica
I20260503 13:57:35.139643 8299 raft_consensus.cc:2243] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.139972 8299 raft_consensus.cc:2272] T a900eebd056343a89cef507a78e29e55 P 1efe002803924b75971d7948c1675b12 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260503 13:57:35.243966 8549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.26.193:46633: connect: Connection refused (error 111)
W20260503 13:57:35.248796 8549 consensus_peers.cc:597] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 -> Peer 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633): Couldn't send request to peer 1efe002803924b75971d7948c1675b12. Status: Network error: Client connection negotiation failed: client connection to 127.8.26.193:46633: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260503 13:57:35.250095 8475 consensus_peers.cc:597] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 -> Peer 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633): Couldn't send request to peer 1efe002803924b75971d7948c1675b12. Status: Network error: Client connection negotiation failed: client connection to 127.8.26.193:46633: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260503 13:57:35.280208 8299 tablet_server.cc:196] TabletServer@127.8.26.193:0 shutdown complete.
I20260503 13:57:35.303158 8299 tablet_server.cc:179] TabletServer@127.8.26.194:0 shutting down...
I20260503 13:57:35.330714 8299 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260503 13:57:35.331475 8299 tablet_replica.cc:333] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7: stopping tablet replica
I20260503 13:57:35.332417 8299 raft_consensus.cc:2243] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.333058 8299 raft_consensus.cc:2272] T 1fe7524a546346368ea1f28ee97c221a P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.334815 8299 tablet_replica.cc:333] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7: stopping tablet replica
I20260503 13:57:35.335249 8299 raft_consensus.cc:2243] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.335644 8299 raft_consensus.cc:2272] T 76a4ada557094372a9d2a6c9b9c24cd6 P a140d8dd75a24d018e082b3ba9603de7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.337301 8299 tablet_replica.cc:333] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7: stopping tablet replica
I20260503 13:57:35.337972 8299 raft_consensus.cc:2243] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.338433 8299 raft_consensus.cc:2272] T a900eebd056343a89cef507a78e29e55 P a140d8dd75a24d018e082b3ba9603de7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.340042 8299 tablet_replica.cc:333] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7: stopping tablet replica
I20260503 13:57:35.340481 8299 raft_consensus.cc:2243] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.340981 8299 raft_consensus.cc:2272] T d7caf0031e354b8ca998e7640505cb9b P a140d8dd75a24d018e082b3ba9603de7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.342530 8299 tablet_replica.cc:333] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7: stopping tablet replica
I20260503 13:57:35.342972 8299 raft_consensus.cc:2243] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [term 1 LEADER]: Raft consensus shutting down.
I20260503 13:57:35.343952 8299 raft_consensus.cc:2272] T 941dbfe411bf460e9a1563b4b946f9a7 P a140d8dd75a24d018e082b3ba9603de7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.345504 8299 tablet_replica.cc:333] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7: stopping tablet replica
I20260503 13:57:35.345952 8299 raft_consensus.cc:2243] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Raft consensus shutting down.
I20260503 13:57:35.347036 8299 raft_consensus.cc:2272] T 9031aeee0f614639943edc1aa0067412 P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.348637 8299 tablet_replica.cc:333] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7: stopping tablet replica
I20260503 13:57:35.349059 8299 raft_consensus.cc:2243] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 3 LEADER]: Raft consensus shutting down.
I20260503 13:57:35.349982 8299 raft_consensus.cc:2272] T cbd326985c85427392b9ecb990c3d888 P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.351462 8299 tablet_replica.cc:333] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7: stopping tablet replica
I20260503 13:57:35.351866 8299 raft_consensus.cc:2243] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.352283 8299 raft_consensus.cc:2272] T 566555790b4d4399bbc0571468916a89 P a140d8dd75a24d018e082b3ba9603de7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.353754 8299 tablet_replica.cc:333] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7: stopping tablet replica
I20260503 13:57:35.354138 8299 raft_consensus.cc:2243] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.354563 8299 raft_consensus.cc:2272] T f2a6dc6f02264540ac18b8a431c42e4a P a140d8dd75a24d018e082b3ba9603de7 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260503 13:57:35.376066 8549 consensus_peers.cc:597] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 -> Peer 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633): Couldn't send request to peer 1efe002803924b75971d7948c1675b12. Status: Network error: Client connection negotiation failed: client connection to 127.8.26.193:46633: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260503 13:57:35.381148 8299 tablet_server.cc:196] TabletServer@127.8.26.194:0 shutdown complete.
I20260503 13:57:35.398165 8299 tablet_server.cc:179] TabletServer@127.8.26.195:0 shutting down...
W20260503 13:57:35.403440 8549 consensus_peers.cc:597] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 -> Peer 1efe002803924b75971d7948c1675b12 (127.8.26.193:46633): Couldn't send request to peer 1efe002803924b75971d7948c1675b12. Status: Network error: Client connection negotiation failed: client connection to 127.8.26.193:46633: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260503 13:57:35.423579 8299 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260503 13:57:35.424208 8299 tablet_replica.cc:333] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61: stopping tablet replica
I20260503 13:57:35.424880 8299 raft_consensus.cc:2243] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [term 2 LEADER]: Raft consensus shutting down.
I20260503 13:57:35.425707 8299 raft_consensus.cc:2272] T 76a4ada557094372a9d2a6c9b9c24cd6 P ff09527976954240a2b7e45977386f61 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.427362 8299 tablet_replica.cc:333] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61: stopping tablet replica
I20260503 13:57:35.427839 8299 raft_consensus.cc:2243] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.428260 8299 raft_consensus.cc:2272] T 1fe7524a546346368ea1f28ee97c221a P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.429804 8299 tablet_replica.cc:333] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61: stopping tablet replica
I20260503 13:57:35.430233 8299 raft_consensus.cc:2243] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.430656 8299 raft_consensus.cc:2272] T f2a6dc6f02264540ac18b8a431c42e4a P ff09527976954240a2b7e45977386f61 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.432158 8299 tablet_replica.cc:333] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61: stopping tablet replica
I20260503 13:57:35.432626 8299 raft_consensus.cc:2243] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 3 LEADER]: Raft consensus shutting down.
I20260503 13:57:35.433553 8299 raft_consensus.cc:2272] T 566555790b4d4399bbc0571468916a89 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.435168 8299 tablet_replica.cc:333] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61: stopping tablet replica
I20260503 13:57:35.435619 8299 raft_consensus.cc:2243] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.436051 8299 raft_consensus.cc:2272] T d7caf0031e354b8ca998e7640505cb9b P ff09527976954240a2b7e45977386f61 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.437593 8299 tablet_replica.cc:333] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61: stopping tablet replica
I20260503 13:57:35.438027 8299 raft_consensus.cc:2243] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.438498 8299 raft_consensus.cc:2272] T 9031aeee0f614639943edc1aa0067412 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.439934 8299 tablet_replica.cc:333] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61: stopping tablet replica
I20260503 13:57:35.440346 8299 raft_consensus.cc:2243] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.440757 8299 raft_consensus.cc:2272] T 941dbfe411bf460e9a1563b4b946f9a7 P ff09527976954240a2b7e45977386f61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.442268 8299 tablet_replica.cc:333] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61: stopping tablet replica
I20260503 13:57:35.442689 8299 raft_consensus.cc:2243] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260503 13:57:35.443099 8299 raft_consensus.cc:2272] T cbd326985c85427392b9ecb990c3d888 P ff09527976954240a2b7e45977386f61 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.444597 8299 tablet_replica.cc:333] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61: stopping tablet replica
I20260503 13:57:35.444988 8299 raft_consensus.cc:2243] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 4 LEADER]: Raft consensus shutting down.
I20260503 13:57:35.445713 8299 raft_consensus.cc:2272] T a900eebd056343a89cef507a78e29e55 P ff09527976954240a2b7e45977386f61 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.471410 8299 tablet_server.cc:196] TabletServer@127.8.26.195:0 shutdown complete.
I20260503 13:57:35.488106 8299 master.cc:562] Master@127.8.26.254:44623 shutting down...
I20260503 13:57:35.617702 8299 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [term 1 LEADER]: Raft consensus shutting down.
I20260503 13:57:35.618374 8299 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260503 13:57:35.618710 8299 tablet_replica.cc:333] T 00000000000000000000000000000000 P b5336cc542d14c8ab2cc89f8c8b9dde2: stopping tablet replica
I20260503 13:57:35.636684 8299 master.cc:584] Master@127.8.26.254:44623 shutdown complete.
1efe002803924b75971d7948c1675b12 4
a140d8dd75a24d018e082b3ba9603de7 4
ff09527976954240a2b7e45977386f61 1
1efe002803924b75971d7948c1675b12 3
a140d8dd75a24d018e082b3ba9603de7 3
ff09527976954240a2b7e45977386f61 3
1efe002803924b75971d7948c1675b12 0
a140d8dd75a24d018e082b3ba9603de7 8
ff09527976954240a2b7e45977386f61 1
1efe002803924b75971d7948c1675b12 3
a140d8dd75a24d018e082b3ba9603de7 3
ff09527976954240a2b7e45977386f61 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30491 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260503 13:57:35.676431 8299 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.8.26.254:39623
I20260503 13:57:35.677568 8299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260503 13:57:35.683636 8822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:57:35.683682 8823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:57:35.684605 8825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260503 13:57:35.685792 8299 server_base.cc:1061] running on GCE node
I20260503 13:57:35.686805 8299 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260503 13:57:35.687036 8299 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260503 13:57:35.687184 8299 hybrid_clock.cc:648] HybridClock initialized: now 1777816655687170 us; error 0 us; skew 500 ppm
I20260503 13:57:35.687696 8299 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260503 13:57:35.690230 8299 webserver.cc:492] Webserver started at http://127.8.26.254:44889/ using document root <none> and password file <none>
I20260503 13:57:35.690663 8299 fs_manager.cc:362] Metadata directory not provided
I20260503 13:57:35.690833 8299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260503 13:57:35.691076 8299 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260503 13:57:35.692212 8299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/master-0-root/instance:
uuid: "2fa153c1e2264aa2ae907ea5273ca055"
format_stamp: "Formatted at 2026-05-03 13:57:35 on dist-test-slave-jcqq"
I20260503 13:57:35.697427 8299 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260503 13:57:35.701551 8830 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:35.702389 8299 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260503 13:57:35.702704 8299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/master-0-root
uuid: "2fa153c1e2264aa2ae907ea5273ca055"
format_stamp: "Formatted at 2026-05-03 13:57:35 on dist-test-slave-jcqq"
I20260503 13:57:35.703058 8299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260503 13:57:35.733034 8299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260503 13:57:35.734261 8299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260503 13:57:35.769255 8299 rpc_server.cc:307] RPC server started. Bound to: 127.8.26.254:39623
I20260503 13:57:35.769352 8881 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.26.254:39623 every 8 connection(s)
I20260503 13:57:35.772840 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:35.781571 8882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055: Bootstrap starting.
I20260503 13:57:35.785890 8882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:35.789625 8882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055: No bootstrap required, opened a new log
I20260503 13:57:35.791558 8882 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa153c1e2264aa2ae907ea5273ca055" member_type: VOTER }
I20260503 13:57:35.791944 8882 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:35.792120 8882 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa153c1e2264aa2ae907ea5273ca055, State: Initialized, Role: FOLLOWER
I20260503 13:57:35.792532 8882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa153c1e2264aa2ae907ea5273ca055" member_type: VOTER }
I20260503 13:57:35.792973 8882 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260503 13:57:35.793164 8882 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260503 13:57:35.793359 8882 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:35.796954 8882 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa153c1e2264aa2ae907ea5273ca055" member_type: VOTER }
I20260503 13:57:35.797376 8882 leader_election.cc:304] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fa153c1e2264aa2ae907ea5273ca055; no voters:
I20260503 13:57:35.798338 8882 leader_election.cc:290] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260503 13:57:35.798702 8885 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:35.799813 8885 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [term 1 LEADER]: Becoming Leader. State: Replica: 2fa153c1e2264aa2ae907ea5273ca055, State: Running, Role: LEADER
I20260503 13:57:35.800436 8885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa153c1e2264aa2ae907ea5273ca055" member_type: VOTER }
I20260503 13:57:35.800890 8882 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [sys.catalog]: configured and running, proceeding with master startup.
I20260503 13:57:35.804859 8886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fa153c1e2264aa2ae907ea5273ca055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa153c1e2264aa2ae907ea5273ca055" member_type: VOTER } }
I20260503 13:57:35.805032 8887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fa153c1e2264aa2ae907ea5273ca055. Latest consensus state: current_term: 1 leader_uuid: "2fa153c1e2264aa2ae907ea5273ca055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa153c1e2264aa2ae907ea5273ca055" member_type: VOTER } }
I20260503 13:57:35.805475 8886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [sys.catalog]: This master's current role is: LEADER
I20260503 13:57:35.805593 8887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [sys.catalog]: This master's current role is: LEADER
I20260503 13:57:35.809957 8894 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260503 13:57:35.816789 8894 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260503 13:57:35.817668 8299 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260503 13:57:35.824597 8894 catalog_manager.cc:1357] Generated new cluster ID: 15453f8e629a4feabbee8f47f6ccf177
I20260503 13:57:35.824851 8894 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260503 13:57:35.848727 8894 catalog_manager.cc:1380] Generated new certificate authority record
I20260503 13:57:35.849864 8894 catalog_manager.cc:1514] Loading token signing keys...
I20260503 13:57:35.862694 8894 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055: Generated new TSK 0
I20260503 13:57:35.863186 8894 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260503 13:57:35.883659 8299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260503 13:57:35.889199 8903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:57:35.889807 8904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260503 13:57:35.891878 8299 server_base.cc:1061] running on GCE node
W20260503 13:57:35.892150 8906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260503 13:57:35.892980 8299 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260503 13:57:35.893167 8299 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260503 13:57:35.893288 8299 hybrid_clock.cc:648] HybridClock initialized: now 1777816655893277 us; error 0 us; skew 500 ppm
I20260503 13:57:35.893877 8299 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260503 13:57:35.895844 8299 webserver.cc:492] Webserver started at http://127.8.26.193:42609/ using document root <none> and password file <none>
I20260503 13:57:35.896246 8299 fs_manager.cc:362] Metadata directory not provided
I20260503 13:57:35.896391 8299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260503 13:57:35.896618 8299 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260503 13:57:35.897549 8299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-0-root/instance:
uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
format_stamp: "Formatted at 2026-05-03 13:57:35 on dist-test-slave-jcqq"
I20260503 13:57:35.901489 8299 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260503 13:57:35.904435 8911 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:35.905189 8299 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260503 13:57:35.905427 8299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-0-root
uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
format_stamp: "Formatted at 2026-05-03 13:57:35 on dist-test-slave-jcqq"
I20260503 13:57:35.905673 8299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260503 13:57:35.925082 8299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260503 13:57:35.926126 8299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260503 13:57:35.927397 8299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260503 13:57:35.929440 8299 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260503 13:57:35.929606 8299 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:35.929780 8299 ts_tablet_manager.cc:616] Registered 0 tablets
I20260503 13:57:35.929909 8299 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:35.965910 8299 rpc_server.cc:307] RPC server started. Bound to: 127.8.26.193:39717
I20260503 13:57:35.966020 8973 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.26.193:39717 every 8 connection(s)
I20260503 13:57:35.970058 8299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260503 13:57:35.977337 8978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:57:35.977705 8979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:57:35.979933 8981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260503 13:57:35.980631 8299 server_base.cc:1061] running on GCE node
I20260503 13:57:35.982056 8299 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260503 13:57:35.982247 8299 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260503 13:57:35.982430 8299 hybrid_clock.cc:648] HybridClock initialized: now 1777816655982412 us; error 0 us; skew 500 ppm
I20260503 13:57:35.982544 8974 heartbeater.cc:344] Connected to a master server at 127.8.26.254:39623
I20260503 13:57:35.982903 8974 heartbeater.cc:461] Registering TS with master...
I20260503 13:57:35.983151 8299 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260503 13:57:35.983688 8974 heartbeater.cc:507] Master 127.8.26.254:39623 requested a full tablet report, sending...
I20260503 13:57:35.985505 8299 webserver.cc:492] Webserver started at http://127.8.26.194:36457/ using document root <none> and password file <none>
I20260503 13:57:35.985836 8847 ts_manager.cc:194] Registered new tserver with Master: aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717)
I20260503 13:57:35.986138 8299 fs_manager.cc:362] Metadata directory not provided
I20260503 13:57:35.986393 8299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260503 13:57:35.986716 8299 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260503 13:57:35.988139 8847 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45886
I20260503 13:57:35.988144 8299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-1-root/instance:
uuid: "1d74da833ad34b2395cda075e2da0c04"
format_stamp: "Formatted at 2026-05-03 13:57:35 on dist-test-slave-jcqq"
I20260503 13:57:35.993026 8299 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260503 13:57:35.996021 8986 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:35.996767 8299 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260503 13:57:35.997020 8299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-1-root
uuid: "1d74da833ad34b2395cda075e2da0c04"
format_stamp: "Formatted at 2026-05-03 13:57:35 on dist-test-slave-jcqq"
I20260503 13:57:35.997251 8299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260503 13:57:36.016805 8299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260503 13:57:36.017921 8299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260503 13:57:36.019213 8299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260503 13:57:36.021332 8299 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260503 13:57:36.021500 8299 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:36.021726 8299 ts_tablet_manager.cc:616] Registered 0 tablets
I20260503 13:57:36.021859 8299 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:36.057116 8299 rpc_server.cc:307] RPC server started. Bound to: 127.8.26.194:37683
I20260503 13:57:36.057204 9048 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.26.194:37683 every 8 connection(s)
I20260503 13:57:36.061340 8299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260503 13:57:36.067600 9052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:57:36.069074 9053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:57:36.071380 9056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260503 13:57:36.072001 9049 heartbeater.cc:344] Connected to a master server at 127.8.26.254:39623
I20260503 13:57:36.072247 8299 server_base.cc:1061] running on GCE node
I20260503 13:57:36.072393 9049 heartbeater.cc:461] Registering TS with master...
I20260503 13:57:36.073148 8299 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260503 13:57:36.073258 9049 heartbeater.cc:507] Master 127.8.26.254:39623 requested a full tablet report, sending...
W20260503 13:57:36.073433 8299 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260503 13:57:36.073665 8299 hybrid_clock.cc:648] HybridClock initialized: now 1777816656073635 us; error 0 us; skew 500 ppm
I20260503 13:57:36.074281 8299 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260503 13:57:36.075153 8847 ts_manager.cc:194] Registered new tserver with Master: 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683)
I20260503 13:57:36.076543 8299 webserver.cc:492] Webserver started at http://127.8.26.195:39399/ using document root <none> and password file <none>
I20260503 13:57:36.076596 8847 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45888
I20260503 13:57:36.077126 8299 fs_manager.cc:362] Metadata directory not provided
I20260503 13:57:36.077328 8299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260503 13:57:36.077612 8299 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260503 13:57:36.078670 8299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-2-root/instance:
uuid: "4f29308110ba4738b29e7adaab74f553"
format_stamp: "Formatted at 2026-05-03 13:57:36 on dist-test-slave-jcqq"
I20260503 13:57:36.082700 8299 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260503 13:57:36.085616 9060 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:36.086280 8299 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260503 13:57:36.086519 8299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-2-root
uuid: "4f29308110ba4738b29e7adaab74f553"
format_stamp: "Formatted at 2026-05-03 13:57:36 on dist-test-slave-jcqq"
I20260503 13:57:36.086752 8299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777816625017654-8299-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260503 13:57:36.114053 8299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260503 13:57:36.115125 8299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260503 13:57:36.116405 8299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260503 13:57:36.120311 8299 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260503 13:57:36.120581 8299 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:36.120846 8299 ts_tablet_manager.cc:616] Registered 0 tablets
I20260503 13:57:36.121053 8299 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260503 13:57:36.157831 8299 rpc_server.cc:307] RPC server started. Bound to: 127.8.26.195:38053
I20260503 13:57:36.157912 9122 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.26.195:38053 every 8 connection(s)
I20260503 13:57:36.169970 9123 heartbeater.cc:344] Connected to a master server at 127.8.26.254:39623
I20260503 13:57:36.170313 9123 heartbeater.cc:461] Registering TS with master...
I20260503 13:57:36.170959 9123 heartbeater.cc:507] Master 127.8.26.254:39623 requested a full tablet report, sending...
I20260503 13:57:36.172689 8847 ts_manager.cc:194] Registered new tserver with Master: 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:36.173718 8299 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013024675s
I20260503 13:57:36.173877 8847 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45892
I20260503 13:57:36.195636 8847 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45904:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260503 13:57:36.198254 8847 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260503 13:57:36.272037 9014 tablet_service.cc:1511] Processing CreateTablet for tablet 804ad412dec7415aaccb82e9bfa87871 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260503 13:57:36.273445 9014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 804ad412dec7415aaccb82e9bfa87871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.273514 8939 tablet_service.cc:1511] Processing CreateTablet for tablet 804ad412dec7415aaccb82e9bfa87871 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260503 13:57:36.274806 8939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 804ad412dec7415aaccb82e9bfa87871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.274519 8934 tablet_service.cc:1511] Processing CreateTablet for tablet 613bed3db7dd46aaa203a8d59930df85 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260503 13:57:36.275722 8934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 613bed3db7dd46aaa203a8d59930df85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.276685 8938 tablet_service.cc:1511] Processing CreateTablet for tablet fc3e5bb9588d4c5c9549b594d40df424 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260503 13:57:36.277916 8938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc3e5bb9588d4c5c9549b594d40df424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.277630 9013 tablet_service.cc:1511] Processing CreateTablet for tablet fc3e5bb9588d4c5c9549b594d40df424 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260503 13:57:36.278857 9013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc3e5bb9588d4c5c9549b594d40df424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.281615 9012 tablet_service.cc:1511] Processing CreateTablet for tablet ee4d02a278c0496db7cfef7d45a5f31c (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260503 13:57:36.282800 8936 tablet_service.cc:1511] Processing CreateTablet for tablet e9d40d8eadbf4810ad68af958e0ffb30 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260503 13:57:36.283977 8936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9d40d8eadbf4810ad68af958e0ffb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.285645 9011 tablet_service.cc:1511] Processing CreateTablet for tablet e9d40d8eadbf4810ad68af958e0ffb30 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260503 13:57:36.285830 8937 tablet_service.cc:1511] Processing CreateTablet for tablet ee4d02a278c0496db7cfef7d45a5f31c (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260503 13:57:36.287007 8937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee4d02a278c0496db7cfef7d45a5f31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.285663 8935 tablet_service.cc:1511] Processing CreateTablet for tablet 8f810c64c8f24fe8a8b85a155f4ce8e4 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260503 13:57:36.287947 8935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f810c64c8f24fe8a8b85a155f4ce8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.284385 9010 tablet_service.cc:1511] Processing CreateTablet for tablet 8f810c64c8f24fe8a8b85a155f4ce8e4 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260503 13:57:36.289700 8933 tablet_service.cc:1511] Processing CreateTablet for tablet 65eae44a7ccd4e7eaf2b1aa1bb415221 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260503 13:57:36.290763 9012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee4d02a278c0496db7cfef7d45a5f31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.290904 8933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65eae44a7ccd4e7eaf2b1aa1bb415221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.293280 9085 tablet_service.cc:1511] Processing CreateTablet for tablet e9d40d8eadbf4810ad68af958e0ffb30 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260503 13:57:36.293988 9087 tablet_service.cc:1511] Processing CreateTablet for tablet fc3e5bb9588d4c5c9549b594d40df424 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260503 13:57:36.294528 9085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9d40d8eadbf4810ad68af958e0ffb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.295256 9087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc3e5bb9588d4c5c9549b594d40df424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.296721 9084 tablet_service.cc:1511] Processing CreateTablet for tablet 8f810c64c8f24fe8a8b85a155f4ce8e4 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260503 13:57:36.297914 9084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f810c64c8f24fe8a8b85a155f4ce8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.297685 9086 tablet_service.cc:1511] Processing CreateTablet for tablet ee4d02a278c0496db7cfef7d45a5f31c (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260503 13:57:36.298921 9086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee4d02a278c0496db7cfef7d45a5f31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.298806 9009 tablet_service.cc:1511] Processing CreateTablet for tablet 613bed3db7dd46aaa203a8d59930df85 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260503 13:57:36.299917 9083 tablet_service.cc:1511] Processing CreateTablet for tablet 613bed3db7dd46aaa203a8d59930df85 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260503 13:57:36.301247 9083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 613bed3db7dd46aaa203a8d59930df85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.297674 9010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f810c64c8f24fe8a8b85a155f4ce8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.302565 9007 tablet_service.cc:1511] Processing CreateTablet for tablet 5df61c04d1414e0f9708d6fc99d92404 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260503 13:57:36.303211 9082 tablet_service.cc:1511] Processing CreateTablet for tablet 65eae44a7ccd4e7eaf2b1aa1bb415221 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260503 13:57:36.303753 9007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5df61c04d1414e0f9708d6fc99d92404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.304869 9011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9d40d8eadbf4810ad68af958e0ffb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.305661 9006 tablet_service.cc:1511] Processing CreateTablet for tablet 31d3ed0d91314eeca257815a1252a43a (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260503 13:57:36.300918 9008 tablet_service.cc:1511] Processing CreateTablet for tablet 65eae44a7ccd4e7eaf2b1aa1bb415221 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260503 13:57:36.306854 9006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d3ed0d91314eeca257815a1252a43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.307883 9082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65eae44a7ccd4e7eaf2b1aa1bb415221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.309940 9088 tablet_service.cc:1511] Processing CreateTablet for tablet 804ad412dec7415aaccb82e9bfa87871 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260503 13:57:36.311074 9088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 804ad412dec7415aaccb82e9bfa87871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.312935 9009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 613bed3db7dd46aaa203a8d59930df85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.316869 9008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65eae44a7ccd4e7eaf2b1aa1bb415221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.317977 9081 tablet_service.cc:1511] Processing CreateTablet for tablet 5df61c04d1414e0f9708d6fc99d92404 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260503 13:57:36.319154 8932 tablet_service.cc:1511] Processing CreateTablet for tablet 5df61c04d1414e0f9708d6fc99d92404 (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260503 13:57:36.320317 9080 tablet_service.cc:1511] Processing CreateTablet for tablet 31d3ed0d91314eeca257815a1252a43a (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260503 13:57:36.319176 9081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5df61c04d1414e0f9708d6fc99d92404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.321667 9080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d3ed0d91314eeca257815a1252a43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.320462 8932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5df61c04d1414e0f9708d6fc99d92404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.322037 8931 tablet_service.cc:1511] Processing CreateTablet for tablet 31d3ed0d91314eeca257815a1252a43a (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260503 13:57:36.323397 8931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d3ed0d91314eeca257815a1252a43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.424644 9005 tablet_service.cc:1511] Processing CreateTablet for tablet cd1db07b9e59487fac793c7ee6364ebc (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260503 13:57:36.425949 9005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd1db07b9e59487fac793c7ee6364ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.426025 8930 tablet_service.cc:1511] Processing CreateTablet for tablet cd1db07b9e59487fac793c7ee6364ebc (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260503 13:57:36.427327 8930 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd1db07b9e59487fac793c7ee6364ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.468045 9138 tablet_bootstrap.cc:492] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb: Bootstrap starting.
I20260503 13:57:36.473939 9079 tablet_service.cc:1511] Processing CreateTablet for tablet cd1db07b9e59487fac793c7ee6364ebc (DEFAULT_TABLE table=test-workload [id=0969ae20bb614c86b5ae6f7603d785bf]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260503 13:57:36.475158 9079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd1db07b9e59487fac793c7ee6364ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:57:36.509287 9139 tablet_bootstrap.cc:492] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04: Bootstrap starting.
I20260503 13:57:36.515053 9139 tablet_bootstrap.cc:654] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.558872 9138 tablet_bootstrap.cc:654] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.563652 9140 tablet_bootstrap.cc:492] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553: Bootstrap starting.
I20260503 13:57:36.565397 9139 tablet_bootstrap.cc:492] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04: No bootstrap required, opened a new log
I20260503 13:57:36.565824 9139 ts_tablet_manager.cc:1403] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.010s
I20260503 13:57:36.568427 9139 raft_consensus.cc:359] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.569082 9139 raft_consensus.cc:385] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.569382 9139 raft_consensus.cc:740] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.569697 9140 tablet_bootstrap.cc:654] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.570097 9139 consensus_queue.cc:260] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.575076 9138 tablet_bootstrap.cc:492] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb: No bootstrap required, opened a new log
I20260503 13:57:36.575526 9138 ts_tablet_manager.cc:1403] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb: Time spent bootstrapping tablet: real 0.108s user 0.005s sys 0.006s
I20260503 13:57:36.577732 9049 heartbeater.cc:499] Master 127.8.26.254:39623 was elected leader, sending a full tablet report...
I20260503 13:57:36.578199 9139 ts_tablet_manager.cc:1434] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260503 13:57:36.578261 9138 raft_consensus.cc:359] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.578990 9138 raft_consensus.cc:385] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.579103 9139 tablet_bootstrap.cc:492] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04: Bootstrap starting.
I20260503 13:57:36.579305 9138 raft_consensus.cc:740] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.580086 9138 consensus_queue.cc:260] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.580813 9140 tablet_bootstrap.cc:492] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553: No bootstrap required, opened a new log
I20260503 13:57:36.581200 9140 ts_tablet_manager.cc:1403] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20260503 13:57:36.583959 9140 raft_consensus.cc:359] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.584640 9140 raft_consensus.cc:385] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.584843 9139 tablet_bootstrap.cc:654] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.584940 9140 raft_consensus.cc:740] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.585930 9140 consensus_queue.cc:260] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.589365 9138 ts_tablet_manager.cc:1434] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260503 13:57:36.590085 8974 heartbeater.cc:499] Master 127.8.26.254:39623 was elected leader, sending a full tablet report...
I20260503 13:57:36.595036 9140 ts_tablet_manager.cc:1434] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
I20260503 13:57:36.595657 9123 heartbeater.cc:499] Master 127.8.26.254:39623 was elected leader, sending a full tablet report...
I20260503 13:57:36.603657 9138 tablet_bootstrap.cc:492] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb: Bootstrap starting.
I20260503 13:57:36.605363 9140 tablet_bootstrap.cc:492] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553: Bootstrap starting.
I20260503 13:57:36.611816 9140 tablet_bootstrap.cc:654] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.613628 9138 tablet_bootstrap.cc:654] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.618686 9139 tablet_bootstrap.cc:492] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04: No bootstrap required, opened a new log
I20260503 13:57:36.619144 9139 ts_tablet_manager.cc:1403] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04: Time spent bootstrapping tablet: real 0.040s user 0.023s sys 0.015s
I20260503 13:57:36.623011 9139 raft_consensus.cc:359] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.623657 9139 raft_consensus.cc:385] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.623939 9139 raft_consensus.cc:740] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.626008 9139 consensus_queue.cc:260] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.627130 9138 tablet_bootstrap.cc:492] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb: No bootstrap required, opened a new log
I20260503 13:57:36.627427 9140 tablet_bootstrap.cc:492] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553: No bootstrap required, opened a new log
I20260503 13:57:36.627573 9138 ts_tablet_manager.cc:1403] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.006s
I20260503 13:57:36.627813 9140 ts_tablet_manager.cc:1403] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.013s
I20260503 13:57:36.627851 9139 ts_tablet_manager.cc:1434] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260503 13:57:36.628790 9139 tablet_bootstrap.cc:492] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04: Bootstrap starting.
I20260503 13:57:36.630252 9138 raft_consensus.cc:359] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.630425 9140 raft_consensus.cc:359] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.630960 9138 raft_consensus.cc:385] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.631063 9140 raft_consensus.cc:385] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.631284 9138 raft_consensus.cc:740] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.631377 9140 raft_consensus.cc:740] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.632062 9140 consensus_queue.cc:260] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.632216 9138 consensus_queue.cc:260] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.634074 9140 ts_tablet_manager.cc:1434] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260503 13:57:36.636726 9140 tablet_bootstrap.cc:492] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553: Bootstrap starting.
I20260503 13:57:36.637964 9139 tablet_bootstrap.cc:654] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.642082 9139 tablet_bootstrap.cc:492] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04: No bootstrap required, opened a new log
I20260503 13:57:36.642417 9139 ts_tablet_manager.cc:1403] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260503 13:57:36.642549 9140 tablet_bootstrap.cc:654] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.642922 9143 raft_consensus.cc:493] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:36.643332 9143 raft_consensus.cc:515] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.645133 9139 raft_consensus.cc:359] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.645762 9143 leader_election.cc:290] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:36.645769 9139 raft_consensus.cc:385] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.646224 9139 raft_consensus.cc:740] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.646965 9139 consensus_queue.cc:260] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.648953 9139 ts_tablet_manager.cc:1434] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260503 13:57:36.649832 9139 tablet_bootstrap.cc:492] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04: Bootstrap starting.
I20260503 13:57:36.651921 9138 ts_tablet_manager.cc:1434] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260503 13:57:36.655581 9139 tablet_bootstrap.cc:654] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.658914 9138 tablet_bootstrap.cc:492] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb: Bootstrap starting.
I20260503 13:57:36.666599 9138 tablet_bootstrap.cc:654] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.666811 9139 tablet_bootstrap.cc:492] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04: No bootstrap required, opened a new log
I20260503 13:57:36.667286 9139 ts_tablet_manager.cc:1403] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.003s
I20260503 13:57:36.670040 9139 raft_consensus.cc:359] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.670715 9139 raft_consensus.cc:385] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.671002 9139 raft_consensus.cc:740] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.671624 9139 consensus_queue.cc:260] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.678140 9138 tablet_bootstrap.cc:492] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb: No bootstrap required, opened a new log
I20260503 13:57:36.678649 9138 ts_tablet_manager.cc:1403] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20260503 13:57:36.680267 9139 ts_tablet_manager.cc:1434] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260503 13:57:36.681615 9138 raft_consensus.cc:359] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.682343 9138 raft_consensus.cc:385] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.682658 9138 raft_consensus.cc:740] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.683447 9138 consensus_queue.cc:260] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.687096 9139 tablet_bootstrap.cc:492] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04: Bootstrap starting.
I20260503 13:57:36.689916 9138 ts_tablet_manager.cc:1434] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20260503 13:57:36.693651 9139 tablet_bootstrap.cc:654] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.696935 9138 tablet_bootstrap.cc:492] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb: Bootstrap starting.
I20260503 13:57:36.703569 9138 tablet_bootstrap.cc:654] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.704173 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553" is_pre_election: true
I20260503 13:57:36.705015 9098 raft_consensus.cc:2468] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 0.
I20260503 13:57:36.706704 8989 leader_election.cc:304] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters:
I20260503 13:57:36.707636 9143 raft_consensus.cc:2804] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:36.707998 9143 raft_consensus.cc:493] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:36.708302 9143 raft_consensus.cc:3060] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:36.715368 9143 raft_consensus.cc:515] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.716809 9138 tablet_bootstrap.cc:492] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb: No bootstrap required, opened a new log
I20260503 13:57:36.717307 9138 ts_tablet_manager.cc:1403] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.009s
I20260503 13:57:36.719570 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:36.720165 9098 raft_consensus.cc:3060] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:36.720055 9138 raft_consensus.cc:359] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.720808 9138 raft_consensus.cc:385] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.721115 9138 raft_consensus.cc:740] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.721976 9138 consensus_queue.cc:260] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.727617 9098 raft_consensus.cc:2468] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 1.
I20260503 13:57:36.735070 8989 leader_election.cc:304] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters:
I20260503 13:57:36.730206 9143 leader_election.cc:290] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 election: Requested vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:36.735985 9143 raft_consensus.cc:2804] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:36.730831 9138 ts_tablet_manager.cc:1434] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260503 13:57:36.733023 8949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" is_pre_election: true
I20260503 13:57:36.733534 8948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
I20260503 13:57:36.736423 9143 raft_consensus.cc:697] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Becoming Leader. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
W20260503 13:57:36.738474 8988 leader_election.cc:343] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:36.739375 9143 consensus_queue.cc:237] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.740068 9145 raft_consensus.cc:493] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:36.740494 9145 raft_consensus.cc:515] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.738447 9138 tablet_bootstrap.cc:492] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb: Bootstrap starting.
I20260503 13:57:36.743017 9145 leader_election.cc:290] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:36.747881 9140 tablet_bootstrap.cc:492] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553: No bootstrap required, opened a new log
I20260503 13:57:36.748245 9138 tablet_bootstrap.cc:654] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.748312 9140 ts_tablet_manager.cc:1403] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553: Time spent bootstrapping tablet: real 0.112s user 0.055s sys 0.042s
I20260503 13:57:36.751071 9140 raft_consensus.cc:359] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.751742 9140 raft_consensus.cc:385] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.752045 9140 raft_consensus.cc:740] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.752743 9140 consensus_queue.cc:260] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
W20260503 13:57:36.740473 8988 leader_election.cc:343] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:36.754703 9139 tablet_bootstrap.cc:492] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04: No bootstrap required, opened a new log
I20260503 13:57:36.755148 9139 ts_tablet_manager.cc:1403] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04: Time spent bootstrapping tablet: real 0.068s user 0.035s sys 0.023s
I20260503 13:57:36.760931 9140 ts_tablet_manager.cc:1434] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553: Time spent starting tablet: real 0.012s user 0.002s sys 0.004s
I20260503 13:57:36.761956 9140 tablet_bootstrap.cc:492] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553: Bootstrap starting.
I20260503 13:57:36.768797 9143 raft_consensus.cc:493] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:36.769726 9143 raft_consensus.cc:515] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.771236 9140 tablet_bootstrap.cc:654] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.771564 9143 leader_election.cc:290] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:36.760967 9139 raft_consensus.cc:359] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.772269 8948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9d40d8eadbf4810ad68af958e0ffb30" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" is_pre_election: true
I20260503 13:57:36.772445 9139 raft_consensus.cc:385] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.772766 9139 raft_consensus.cc:740] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Initialized, Role: FOLLOWER
W20260503 13:57:36.773710 8988 leader_election.cc:343] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:36.778031 8844 catalog_manager.cc:5671] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194). New cstate: current_term: 1 leader_uuid: "1d74da833ad34b2395cda075e2da0c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:36.773473 9139 consensus_queue.cc:260] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
W20260503 13:57:36.780838 8989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:38053, user_credentials={real_user=slave}} blocked reactor thread for 45864.8us
I20260503 13:57:36.782816 9139 ts_tablet_manager.cc:1434] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20260503 13:57:36.784024 9146 raft_consensus.cc:493] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:36.789477 9138 tablet_bootstrap.cc:492] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb: No bootstrap required, opened a new log
I20260503 13:57:36.789491 9146 raft_consensus.cc:515] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.790059 9138 ts_tablet_manager.cc:1403] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb: Time spent bootstrapping tablet: real 0.052s user 0.016s sys 0.014s
I20260503 13:57:36.791955 9146 leader_election.cc:290] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683)
I20260503 13:57:36.792927 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd1db07b9e59487fac793c7ee6364ebc" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553" is_pre_election: true
I20260503 13:57:36.793426 9139 tablet_bootstrap.cc:492] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04: Bootstrap starting.
I20260503 13:57:36.797312 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd1db07b9e59487fac793c7ee6364ebc" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04" is_pre_election: true
I20260503 13:57:36.797935 9024 raft_consensus.cc:2468] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 0.
I20260503 13:57:36.798378 9138 raft_consensus.cc:359] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.799247 8915 leader_election.cc:304] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:36.800155 9145 raft_consensus.cc:2804] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:36.800498 9145 raft_consensus.cc:493] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:36.803901 9145 raft_consensus.cc:3060] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:36.799273 9138 raft_consensus.cc:385] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.806558 9138 raft_consensus.cc:740] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.806614 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9d40d8eadbf4810ad68af958e0ffb30" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553" is_pre_election: true
I20260503 13:57:36.807298 9138 consensus_queue.cc:260] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
W20260503 13:57:36.807837 8989 leader_election.cc:343] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:36.805541 9139 tablet_bootstrap.cc:654] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.808249 8989 leader_election.cc:304] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04; no voters: 4f29308110ba4738b29e7adaab74f553, aea6a76deb4a4551af32917c5b8ff3eb
I20260503 13:57:36.810745 9158 raft_consensus.cc:2749] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260503 13:57:36.811786 8895 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:57:36.816332 9140 tablet_bootstrap.cc:492] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553: No bootstrap required, opened a new log
I20260503 13:57:36.816880 9140 ts_tablet_manager.cc:1403] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.000s
I20260503 13:57:36.820015 9140 raft_consensus.cc:359] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.820760 9140 raft_consensus.cc:385] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.821216 9140 raft_consensus.cc:740] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.821411 9145 raft_consensus.cc:515] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.824458 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd1db07b9e59487fac793c7ee6364ebc" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:36.825105 9024 raft_consensus.cc:3060] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:57:36.831646 8914 leader_election.cc:343] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:36.823634 9145 leader_election.cc:290] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 election: Requested vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:36.832804 9162 raft_consensus.cc:493] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:36.835862 9162 raft_consensus.cc:515] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.838883 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65eae44a7ccd4e7eaf2b1aa1bb415221" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553" is_pre_election: true
I20260503 13:57:36.839453 9098 raft_consensus.cc:2468] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 0.
I20260503 13:57:36.839975 8948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65eae44a7ccd4e7eaf2b1aa1bb415221" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" is_pre_election: true
I20260503 13:57:36.840601 8989 leader_election.cc:304] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters:
I20260503 13:57:36.841421 9158 raft_consensus.cc:2804] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:36.841740 9158 raft_consensus.cc:493] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:36.842026 9158 raft_consensus.cc:3060] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:36.843608 9024 raft_consensus.cc:2468] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 1.
I20260503 13:57:36.843822 9138 ts_tablet_manager.cc:1434] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb: Time spent starting tablet: real 0.053s user 0.007s sys 0.010s
I20260503 13:57:36.844841 9138 tablet_bootstrap.cc:492] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb: Bootstrap starting.
I20260503 13:57:36.845258 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd1db07b9e59487fac793c7ee6364ebc" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:36.840610 8948 raft_consensus.cc:2468] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 0.
I20260503 13:57:36.848471 9158 raft_consensus.cc:515] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.850613 9138 tablet_bootstrap.cc:654] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.851493 9023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc3e5bb9588d4c5c9549b594d40df424" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04" is_pre_election: true
I20260503 13:57:36.851943 8948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65eae44a7ccd4e7eaf2b1aa1bb415221" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
I20260503 13:57:36.852490 8948 raft_consensus.cc:3060] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:36.853629 9158 leader_election.cc:290] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 election: Requested vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:36.853996 9097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65eae44a7ccd4e7eaf2b1aa1bb415221" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:36.854048 9162 leader_election.cc:290] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:36.854696 9097 raft_consensus.cc:3060] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:57:36.855369 9064 leader_election.cc:343] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:36.858575 9139 tablet_bootstrap.cc:492] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04: No bootstrap required, opened a new log
I20260503 13:57:36.858947 8948 raft_consensus.cc:2468] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 1.
I20260503 13:57:36.859117 9139 ts_tablet_manager.cc:1403] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.011s
I20260503 13:57:36.859954 8988 leader_election.cc:304] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:36.860773 9162 raft_consensus.cc:2804] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:36.861202 9162 raft_consensus.cc:697] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Becoming Leader. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
I20260503 13:57:36.861320 9097 raft_consensus.cc:2468] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 1.
I20260503 13:57:36.823077 9140 consensus_queue.cc:260] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.862673 9098 raft_consensus.cc:3060] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:36.862002 9162 consensus_queue.cc:237] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.864380 9139 raft_consensus.cc:359] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.865844 8915 leader_election.cc:304] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:36.866565 9139 raft_consensus.cc:385] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.866942 9139 raft_consensus.cc:740] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.867002 9168 raft_consensus.cc:2804] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:36.867584 9168 raft_consensus.cc:697] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Becoming Leader. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:36.867801 9139 consensus_queue.cc:260] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.868773 9098 raft_consensus.cc:2468] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 1.
I20260503 13:57:36.868394 9168 consensus_queue.cc:237] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.870630 9139 ts_tablet_manager.cc:1434] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20260503 13:57:36.871712 9139 tablet_bootstrap.cc:492] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04: Bootstrap starting.
I20260503 13:57:36.866039 9140 ts_tablet_manager.cc:1434] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553: Time spent starting tablet: real 0.049s user 0.006s sys 0.001s
I20260503 13:57:36.882426 9140 tablet_bootstrap.cc:492] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553: Bootstrap starting.
I20260503 13:57:36.885900 9138 tablet_bootstrap.cc:492] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb: No bootstrap required, opened a new log
I20260503 13:57:36.886415 9138 ts_tablet_manager.cc:1403] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb: Time spent bootstrapping tablet: real 0.042s user 0.016s sys 0.011s
I20260503 13:57:36.886458 8948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc3e5bb9588d4c5c9549b594d40df424" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" is_pre_election: true
W20260503 13:57:36.888228 9062 leader_election.cc:343] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:36.888589 9140 tablet_bootstrap.cc:654] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.888654 9062 leader_election.cc:304] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f29308110ba4738b29e7adaab74f553; no voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb
I20260503 13:57:36.890583 9138 raft_consensus.cc:359] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.890093 9146 raft_consensus.cc:2749] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:57:36.895700 9139 tablet_bootstrap.cc:654] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.891880 9138 raft_consensus.cc:385] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.896351 9138 raft_consensus.cc:740] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.896940 9138 consensus_queue.cc:260] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.898396 9140 tablet_bootstrap.cc:492] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553: No bootstrap required, opened a new log
I20260503 13:57:36.898818 9140 ts_tablet_manager.cc:1403] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20260503 13:57:36.899154 9138 ts_tablet_manager.cc:1434] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260503 13:57:36.900048 9138 tablet_bootstrap.cc:492] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb: Bootstrap starting.
I20260503 13:57:36.901381 9140 raft_consensus.cc:359] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.902033 9140 raft_consensus.cc:385] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.902304 9140 raft_consensus.cc:740] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.902969 9140 consensus_queue.cc:260] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.905296 9140 ts_tablet_manager.cc:1434] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260503 13:57:36.906205 9140 tablet_bootstrap.cc:492] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553: Bootstrap starting.
I20260503 13:57:36.908797 9138 tablet_bootstrap.cc:654] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.911259 8845 catalog_manager.cc:5671] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb reported cstate change: term changed from 0 to 1, leader changed from <none> to aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193). New cstate: current_term: 1 leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:36.912117 9140 tablet_bootstrap.cc:654] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.915371 8844 catalog_manager.cc:5671] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194). New cstate: current_term: 1 leader_uuid: "1d74da833ad34b2395cda075e2da0c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:36.917402 9139 tablet_bootstrap.cc:492] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04: No bootstrap required, opened a new log
I20260503 13:57:36.917804 9140 tablet_bootstrap.cc:492] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553: No bootstrap required, opened a new log
I20260503 13:57:36.917868 9139 ts_tablet_manager.cc:1403] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.000s
I20260503 13:57:36.918332 9140 ts_tablet_manager.cc:1403] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260503 13:57:36.920856 9139 raft_consensus.cc:359] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.921172 9140 raft_consensus.cc:359] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.921574 9139 raft_consensus.cc:385] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.921875 9140 raft_consensus.cc:385] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.921994 9139 raft_consensus.cc:740] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.922286 9140 raft_consensus.cc:740] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.923007 9139 consensus_queue.cc:260] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.923125 9140 consensus_queue.cc:260] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.925233 9139 ts_tablet_manager.cc:1434] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260503 13:57:36.925590 9140 ts_tablet_manager.cc:1434] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260503 13:57:36.926231 9139 tablet_bootstrap.cc:492] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04: Bootstrap starting.
I20260503 13:57:36.926554 9140 tablet_bootstrap.cc:492] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553: Bootstrap starting.
I20260503 13:57:36.932629 9139 tablet_bootstrap.cc:654] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.932911 9140 tablet_bootstrap.cc:654] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.938963 9140 tablet_bootstrap.cc:492] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553: No bootstrap required, opened a new log
I20260503 13:57:36.939424 9140 ts_tablet_manager.cc:1403] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260503 13:57:36.945190 9138 tablet_bootstrap.cc:492] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb: No bootstrap required, opened a new log
I20260503 13:57:36.945621 9138 ts_tablet_manager.cc:1403] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb: Time spent bootstrapping tablet: real 0.046s user 0.023s sys 0.015s
I20260503 13:57:36.946774 9139 tablet_bootstrap.cc:492] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04: No bootstrap required, opened a new log
I20260503 13:57:36.947184 9139 ts_tablet_manager.cc:1403] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.012s
I20260503 13:57:36.948371 9138 raft_consensus.cc:359] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.949162 9138 raft_consensus.cc:385] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.944790 9140 raft_consensus.cc:359] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.949429 9138 raft_consensus.cc:740] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.949558 9140 raft_consensus.cc:385] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.949874 9140 raft_consensus.cc:740] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.950136 9138 consensus_queue.cc:260] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.949432 9139 raft_consensus.cc:359] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.951367 9139 raft_consensus.cc:385] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.950593 9140 consensus_queue.cc:260] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.951610 9139 raft_consensus.cc:740] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.952209 9138 ts_tablet_manager.cc:1434] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260503 13:57:36.952296 9139 consensus_queue.cc:260] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.953073 9140 ts_tablet_manager.cc:1434] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553: Time spent starting tablet: real 0.013s user 0.006s sys 0.002s
I20260503 13:57:36.953083 9162 raft_consensus.cc:493] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:36.953392 9138 tablet_bootstrap.cc:492] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb: Bootstrap starting.
I20260503 13:57:36.954046 9140 tablet_bootstrap.cc:492] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553: Bootstrap starting.
I20260503 13:57:36.953673 9162 raft_consensus.cc:515] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.955579 9162 leader_election.cc:290] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:36.956987 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553" is_pre_election: true
I20260503 13:57:36.957520 9098 raft_consensus.cc:2468] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 0.
I20260503 13:57:36.958861 8989 leader_election.cc:304] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters:
I20260503 13:57:36.959633 9138 tablet_bootstrap.cc:654] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.959600 9162 raft_consensus.cc:2804] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:36.960078 9140 tablet_bootstrap.cc:654] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.960052 9162 raft_consensus.cc:493] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:36.960512 9162 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:36.961949 8948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" is_pre_election: true
W20260503 13:57:36.964031 8988 leader_election.cc:343] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:36.967749 9162 raft_consensus.cc:515] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.970206 8948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
W20260503 13:57:36.971372 8988 leader_election.cc:343] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:36.973124 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:36.973623 9098 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:36.977371 9139 ts_tablet_manager.cc:1434] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260503 13:57:36.978276 9139 tablet_bootstrap.cc:492] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04: Bootstrap starting.
I20260503 13:57:36.982442 9098 raft_consensus.cc:2468] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 1.
I20260503 13:57:36.983551 9139 tablet_bootstrap.cc:654] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:36.983484 8989 leader_election.cc:304] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters: aea6a76deb4a4551af32917c5b8ff3eb
I20260503 13:57:36.984313 9158 raft_consensus.cc:2804] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:36.984773 9158 raft_consensus.cc:697] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Becoming Leader. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
I20260503 13:57:36.985517 9158 consensus_queue.cc:237] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.994818 9139 tablet_bootstrap.cc:492] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04: No bootstrap required, opened a new log
I20260503 13:57:36.995338 9139 ts_tablet_manager.cc:1403] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20260503 13:57:36.997884 9139 raft_consensus.cc:359] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.998946 9139 raft_consensus.cc:385] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:36.999282 9139 raft_consensus.cc:740] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Initialized, Role: FOLLOWER
I20260503 13:57:36.999917 9139 consensus_queue.cc:260] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:36.994818 9162 leader_election.cc:290] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 election: Requested vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.002506 9139 ts_tablet_manager.cc:1434] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260503 13:57:37.003423 9139 tablet_bootstrap.cc:492] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04: Bootstrap starting.
I20260503 13:57:37.004467 9168 raft_consensus.cc:493] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:37.004968 9168 raft_consensus.cc:515] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.005275 8847 catalog_manager.cc:5671] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194). New cstate: current_term: 1 leader_uuid: "1d74da833ad34b2395cda075e2da0c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:37.006953 9168 leader_election.cc:290] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.009052 9139 tablet_bootstrap.cc:654] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:37.009347 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d3ed0d91314eeca257815a1252a43a" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553" is_pre_election: true
I20260503 13:57:37.010499 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d3ed0d91314eeca257815a1252a43a" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04" is_pre_election: true
I20260503 13:57:37.011121 9024 raft_consensus.cc:2468] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 0.
I20260503 13:57:37.015887 9168 raft_consensus.cc:493] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:37.016345 9168 raft_consensus.cc:515] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.018257 9168 leader_election.cc:290] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
W20260503 13:57:37.010589 8914 leader_election.cc:343] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:37.019951 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553" is_pre_election: true
I20260503 13:57:37.020617 9098 raft_consensus.cc:2468] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 0.
I20260503 13:57:37.021742 8914 leader_election.cc:304] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f29308110ba4738b29e7adaab74f553, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:37.021718 8915 leader_election.cc:304] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb; no voters: 4f29308110ba4738b29e7adaab74f553
I20260503 13:57:37.022720 9168 raft_consensus.cc:2804] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:37.023124 9168 raft_consensus.cc:493] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:37.023196 9145 raft_consensus.cc:2804] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:37.023507 9168 raft_consensus.cc:3060] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.023618 9145 raft_consensus.cc:493] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:37.024034 9145 raft_consensus.cc:3060] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.030159 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04" is_pre_election: true
I20260503 13:57:37.030754 9024 raft_consensus.cc:2468] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 0.
I20260503 13:57:37.031005 9168 raft_consensus.cc:515] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.033776 9023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:37.034062 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:37.034636 9098 raft_consensus.cc:3060] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.037114 9145 raft_consensus.cc:515] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.039673 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d3ed0d91314eeca257815a1252a43a" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:37.040211 9024 raft_consensus.cc:3060] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.041538 9098 raft_consensus.cc:2468] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 1.
I20260503 13:57:37.042629 8914 leader_election.cc:304] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4f29308110ba4738b29e7adaab74f553, aea6a76deb4a4551af32917c5b8ff3eb; no voters: 1d74da833ad34b2395cda075e2da0c04
I20260503 13:57:37.046821 9024 raft_consensus.cc:2468] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 1.
I20260503 13:57:37.047871 8915 leader_election.cc:304] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:37.053850 9168 leader_election.cc:290] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 election: Requested vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.054426 9168 raft_consensus.cc:2804] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:37.054850 9168 raft_consensus.cc:697] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Becoming Leader. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:37.055366 9145 leader_election.cc:290] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 election: Requested vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.055860 9145 raft_consensus.cc:2804] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:37.055644 9168 consensus_queue.cc:237] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.056254 9145 raft_consensus.cc:697] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Becoming Leader. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:37.060061 9145 consensus_queue.cc:237] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.067559 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d3ed0d91314eeca257815a1252a43a" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553"
W20260503 13:57:37.068769 8914 leader_election.cc:343] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:37.070834 9138 tablet_bootstrap.cc:492] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb: No bootstrap required, opened a new log
I20260503 13:57:37.071095 9140 tablet_bootstrap.cc:492] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553: No bootstrap required, opened a new log
I20260503 13:57:37.071270 9138 ts_tablet_manager.cc:1403] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb: Time spent bootstrapping tablet: real 0.118s user 0.052s sys 0.055s
I20260503 13:57:37.071462 9140 ts_tablet_manager.cc:1403] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553: Time spent bootstrapping tablet: real 0.118s user 0.028s sys 0.038s
I20260503 13:57:37.072099 9139 tablet_bootstrap.cc:492] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04: No bootstrap required, opened a new log
I20260503 13:57:37.072420 9139 ts_tablet_manager.cc:1403] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04: Time spent bootstrapping tablet: real 0.069s user 0.023s sys 0.038s
I20260503 13:57:37.073722 9138 raft_consensus.cc:359] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.073911 9140 raft_consensus.cc:359] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.074352 9138 raft_consensus.cc:385] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:37.074465 9140 raft_consensus.cc:385] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:37.074165 9139 raft_consensus.cc:359] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.074687 9138 raft_consensus.cc:740] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Initialized, Role: FOLLOWER
I20260503 13:57:37.074750 9140 raft_consensus.cc:740] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Initialized, Role: FOLLOWER
I20260503 13:57:37.074857 9139 raft_consensus.cc:385] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:37.075374 9139 raft_consensus.cc:740] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Initialized, Role: FOLLOWER
I20260503 13:57:37.075575 9140 consensus_queue.cc:260] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.075623 9138 consensus_queue.cc:260] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.077837 9138 ts_tablet_manager.cc:1434] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260503 13:57:37.078392 9140 ts_tablet_manager.cc:1434] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260503 13:57:37.078691 9138 tablet_bootstrap.cc:492] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb: Bootstrap starting.
I20260503 13:57:37.079192 9140 tablet_bootstrap.cc:492] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553: Bootstrap starting.
I20260503 13:57:37.082782 9145 raft_consensus.cc:493] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:37.083240 9145 raft_consensus.cc:515] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.084769 9140 tablet_bootstrap.cc:654] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:37.085322 9145 leader_election.cc:290] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.085996 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f810c64c8f24fe8a8b85a155f4ce8e4" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04" is_pre_election: true
I20260503 13:57:37.086552 9024 raft_consensus.cc:2468] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 0.
I20260503 13:57:37.086817 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f810c64c8f24fe8a8b85a155f4ce8e4" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553" is_pre_election: true
I20260503 13:57:37.087388 9098 raft_consensus.cc:2468] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 0.
I20260503 13:57:37.088028 9138 tablet_bootstrap.cc:654] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:37.088037 8915 leader_election.cc:304] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:37.088939 9145 raft_consensus.cc:2804] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:37.089303 9145 raft_consensus.cc:493] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:37.089617 9145 raft_consensus.cc:3060] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.076188 9139 consensus_queue.cc:260] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.094272 9139 ts_tablet_manager.cc:1434] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20260503 13:57:37.095151 9140 tablet_bootstrap.cc:492] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553: No bootstrap required, opened a new log
I20260503 13:57:37.095556 9140 ts_tablet_manager.cc:1403] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20260503 13:57:37.096258 9145 raft_consensus.cc:515] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.098011 9140 raft_consensus.cc:359] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.098587 9140 raft_consensus.cc:385] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:37.098711 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f810c64c8f24fe8a8b85a155f4ce8e4" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:37.098959 9140 raft_consensus.cc:740] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Initialized, Role: FOLLOWER
I20260503 13:57:37.099241 9024 raft_consensus.cc:3060] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.099694 9140 consensus_queue.cc:260] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.100461 9145 leader_election.cc:290] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 election: Requested vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.101601 9140 ts_tablet_manager.cc:1434] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260503 13:57:37.102643 9140 tablet_bootstrap.cc:492] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553: Bootstrap starting.
I20260503 13:57:37.102720 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f810c64c8f24fe8a8b85a155f4ce8e4" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:37.103286 9098 raft_consensus.cc:3060] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.108803 9140 tablet_bootstrap.cc:654] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:37.114516 9024 raft_consensus.cc:2468] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 1.
I20260503 13:57:37.115198 9158 raft_consensus.cc:493] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:37.115692 8915 leader_election.cc:304] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:37.115623 9158 raft_consensus.cc:515] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.116619 9168 raft_consensus.cc:2804] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:37.117435 9168 raft_consensus.cc:697] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Becoming Leader. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:37.119007 9168 consensus_queue.cc:237] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.119658 9138 tablet_bootstrap.cc:492] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb: No bootstrap required, opened a new log
I20260503 13:57:37.119730 9097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc3e5bb9588d4c5c9549b594d40df424" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553" is_pre_election: true
I20260503 13:57:37.120182 9138 ts_tablet_manager.cc:1403] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb: Time spent bootstrapping tablet: real 0.042s user 0.014s sys 0.009s
I20260503 13:57:37.120354 9097 raft_consensus.cc:2468] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 0.
I20260503 13:57:37.119033 8948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc3e5bb9588d4c5c9549b594d40df424" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" is_pre_election: true
W20260503 13:57:37.122359 8988 leader_election.cc:343] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:57:37.114667 8847 catalog_manager.cc:5671] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb reported cstate change: term changed from 0 to 1, leader changed from <none> to aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193). New cstate: current_term: 1 leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:37.122924 8989 leader_election.cc:304] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters: aea6a76deb4a4551af32917c5b8ff3eb
I20260503 13:57:37.122865 9138 raft_consensus.cc:359] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.123646 9138 raft_consensus.cc:385] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:37.123786 9162 raft_consensus.cc:2804] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:37.123924 9138 raft_consensus.cc:740] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Initialized, Role: FOLLOWER
I20260503 13:57:37.117967 9158 leader_election.cc:290] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.124222 9162 raft_consensus.cc:493] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:37.124604 9162 raft_consensus.cc:3060] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.125347 9098 raft_consensus.cc:2468] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 1.
I20260503 13:57:37.125131 9138 consensus_queue.cc:260] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.127383 9138 ts_tablet_manager.cc:1434] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260503 13:57:37.128278 9138 tablet_bootstrap.cc:492] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb: Bootstrap starting.
I20260503 13:57:37.123967 8847 catalog_manager.cc:5671] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb reported cstate change: term changed from 0 to 1, leader changed from <none> to aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193). New cstate: current_term: 1 leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:37.134464 9138 tablet_bootstrap.cc:654] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb: Neither blocks nor log segments found. Creating new log.
I20260503 13:57:37.134476 9162 raft_consensus.cc:515] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.139722 9162 leader_election.cc:290] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 election: Requested vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.141317 8948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc3e5bb9588d4c5c9549b594d40df424" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
I20260503 13:57:37.141868 8948 raft_consensus.cc:3060] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.142709 9140 tablet_bootstrap.cc:492] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553: No bootstrap required, opened a new log
I20260503 13:57:37.143164 9140 ts_tablet_manager.cc:1403] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.020s
I20260503 13:57:37.145402 9138 tablet_bootstrap.cc:492] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb: No bootstrap required, opened a new log
I20260503 13:57:37.145805 9138 ts_tablet_manager.cc:1403] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260503 13:57:37.145645 9140 raft_consensus.cc:359] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.146242 9140 raft_consensus.cc:385] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:37.146216 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc3e5bb9588d4c5c9549b594d40df424" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:37.146651 9140 raft_consensus.cc:740] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Initialized, Role: FOLLOWER
I20260503 13:57:37.146862 9098 raft_consensus.cc:3060] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.147372 9140 consensus_queue.cc:260] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.148218 9138 raft_consensus.cc:359] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.148736 8948 raft_consensus.cc:2468] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 1.
I20260503 13:57:37.148974 9138 raft_consensus.cc:385] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:57:37.149278 9138 raft_consensus.cc:740] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Initialized, Role: FOLLOWER
I20260503 13:57:37.150008 8988 leader_election.cc:304] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:37.149930 9138 consensus_queue.cc:260] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.151391 9140 ts_tablet_manager.cc:1434] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260503 13:57:37.151753 9138 ts_tablet_manager.cc:1434] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260503 13:57:37.154786 9162 raft_consensus.cc:2804] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:37.155409 9098 raft_consensus.cc:2468] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 1.
I20260503 13:57:37.155244 9162 raft_consensus.cc:697] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Becoming Leader. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
I20260503 13:57:37.156533 9162 consensus_queue.cc:237] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.159520 8846 catalog_manager.cc:5671] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb reported cstate change: term changed from 0 to 1, leader changed from <none> to aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193). New cstate: current_term: 1 leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:37.169051 8847 catalog_manager.cc:5671] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194). New cstate: current_term: 1 leader_uuid: "1d74da833ad34b2395cda075e2da0c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:37.172833 9146 raft_consensus.cc:493] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:37.173298 9146 raft_consensus.cc:515] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.174764 9146 leader_election.cc:290] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683)
I20260503 13:57:37.175272 8948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" is_pre_election: true
I20260503 13:57:37.175815 8948 raft_consensus.cc:2468] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f29308110ba4738b29e7adaab74f553 in term 0.
I20260503 13:57:37.176119 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04" is_pre_election: true
I20260503 13:57:37.176676 9024 raft_consensus.cc:2468] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f29308110ba4738b29e7adaab74f553 in term 0.
I20260503 13:57:37.177086 9062 leader_election.cc:304] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f29308110ba4738b29e7adaab74f553, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:37.177745 9146 raft_consensus.cc:2804] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:37.178040 9146 raft_consensus.cc:493] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:37.178274 9146 raft_consensus.cc:3060] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.182495 9146 raft_consensus.cc:515] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.183883 9146 leader_election.cc:290] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 1 election: Requested vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683)
I20260503 13:57:37.184619 8948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
I20260503 13:57:37.184890 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:37.185075 8948 raft_consensus.cc:3060] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.185348 9024 raft_consensus.cc:3060] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.189239 8948 raft_consensus.cc:2468] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f29308110ba4738b29e7adaab74f553 in term 1.
I20260503 13:57:37.189728 9024 raft_consensus.cc:2468] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f29308110ba4738b29e7adaab74f553 in term 1.
I20260503 13:57:37.190013 9062 leader_election.cc:304] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f29308110ba4738b29e7adaab74f553, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:37.190652 9146 raft_consensus.cc:2804] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:37.191459 9146 raft_consensus.cc:697] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 1 LEADER]: Becoming Leader. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Running, Role: LEADER
I20260503 13:57:37.192063 9146 consensus_queue.cc:237] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.194136 8948 raft_consensus.cc:3060] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.195115 9168 raft_consensus.cc:493] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:37.195586 9168 raft_consensus.cc:515] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.197365 9168 leader_election.cc:290] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.198421 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9d40d8eadbf4810ad68af958e0ffb30" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04" is_pre_election: true
I20260503 13:57:37.198787 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9d40d8eadbf4810ad68af958e0ffb30" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553" is_pre_election: true
I20260503 13:57:37.199247 9024 raft_consensus.cc:2468] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 0.
I20260503 13:57:37.199748 9168 raft_consensus.cc:493] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:37.200162 9168 raft_consensus.cc:515] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.199854 8847 catalog_manager.cc:5671] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f29308110ba4738b29e7adaab74f553 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4f29308110ba4738b29e7adaab74f553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: HEALTHY } } }
I20260503 13:57:37.199317 9098 raft_consensus.cc:2468] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 0.
I20260503 13:57:37.200335 8915 leader_election.cc:304] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:37.201289 9162 consensus_queue.cc:1048] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260503 13:57:37.202116 9145 raft_consensus.cc:2804] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:57:37.202471 9145 raft_consensus.cc:493] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:57:37.202798 9145 raft_consensus.cc:3060] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.203640 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553" is_pre_election: true
I20260503 13:57:37.204339 9098 raft_consensus.cc:2393] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aea6a76deb4a4551af32917c5b8ff3eb in current term 1: Already voted for candidate 1d74da833ad34b2395cda075e2da0c04 in this term.
I20260503 13:57:37.206817 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04" is_pre_election: true
I20260503 13:57:37.208312 8915 leader_election.cc:304] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea6a76deb4a4551af32917c5b8ff3eb; no voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553
I20260503 13:57:37.209209 9187 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.210492 9145 raft_consensus.cc:515] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.211774 9145 leader_election.cc:290] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 election: Requested vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.212952 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9d40d8eadbf4810ad68af958e0ffb30" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:37.213922 9098 raft_consensus.cc:3060] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.214345 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9d40d8eadbf4810ad68af958e0ffb30" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:37.214943 9024 raft_consensus.cc:3060] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.218042 9187 raft_consensus.cc:2749] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:57:37.219472 9162 consensus_queue.cc:1048] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.220511 9098 raft_consensus.cc:2468] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 1.
I20260503 13:57:37.221570 8914 leader_election.cc:304] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f29308110ba4738b29e7adaab74f553, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:37.222435 9187 raft_consensus.cc:2804] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:57:37.214479 9168 leader_election.cc:290] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.223093 9187 raft_consensus.cc:697] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Becoming Leader. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:37.223955 9187 consensus_queue.cc:237] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.225711 9024 raft_consensus.cc:2468] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 1.
I20260503 13:57:37.257545 8846 catalog_manager.cc:5671] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb reported cstate change: term changed from 0 to 1, leader changed from <none> to aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193). New cstate: current_term: 1 leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:37.271785 9187 consensus_queue.cc:1048] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.281759 9168 consensus_queue.cc:1048] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.369844 9024 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871"
dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
mode: GRACEFUL
new_leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:45812
I20260503 13:57:37.370337 9024 raft_consensus.cc:606] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Received request to transfer leadership to TS aea6a76deb4a4551af32917c5b8ff3eb
I20260503 13:57:37.374567 9024 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fc3e5bb9588d4c5c9549b594d40df424"
dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
mode: GRACEFUL
new_leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:45812
I20260503 13:57:37.374995 9024 raft_consensus.cc:606] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Received request to transfer leadership to TS aea6a76deb4a4551af32917c5b8ff3eb
I20260503 13:57:37.378901 9024 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c"
dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
mode: GRACEFUL
new_leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:45812
I20260503 13:57:37.379317 9024 raft_consensus.cc:606] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Received request to transfer leadership to TS aea6a76deb4a4551af32917c5b8ff3eb
I20260503 13:57:37.394343 9158 consensus_queue.cc:1048] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.394879 8948 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
mode: GRACEFUL
new_leader_uuid: "1d74da833ad34b2395cda075e2da0c04"
from {username='slave'} at 127.0.0.1:56290
I20260503 13:57:37.395437 8948 raft_consensus.cc:606] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Received request to transfer leadership to TS 1d74da833ad34b2395cda075e2da0c04
I20260503 13:57:37.405620 9143 consensus_queue.cc:1048] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.419833 9146 raft_consensus.cc:493] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:57:37.420305 9146 raft_consensus.cc:515] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.422158 9146 leader_election.cc:290] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683)
I20260503 13:57:37.423074 8948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d3ed0d91314eeca257815a1252a43a" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" is_pre_election: true
I20260503 13:57:37.423274 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d3ed0d91314eeca257815a1252a43a" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d74da833ad34b2395cda075e2da0c04" is_pre_election: true
I20260503 13:57:37.423945 9024 raft_consensus.cc:2393] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4f29308110ba4738b29e7adaab74f553 in current term 1: Already voted for candidate aea6a76deb4a4551af32917c5b8ff3eb in this term.
I20260503 13:57:37.424930 9064 leader_election.cc:304] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f29308110ba4738b29e7adaab74f553; no voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb
I20260503 13:57:37.425539 9146 raft_consensus.cc:3060] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.427764 9097 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404"
dest_uuid: "4f29308110ba4738b29e7adaab74f553"
mode: GRACEFUL
new_leader_uuid: "1d74da833ad34b2395cda075e2da0c04"
from {username='slave'} at 127.0.0.1:53726
I20260503 13:57:37.428328 9097 raft_consensus.cc:606] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 1 LEADER]: Received request to transfer leadership to TS 1d74da833ad34b2395cda075e2da0c04
I20260503 13:57:37.431097 9146 raft_consensus.cc:2749] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:57:37.433094 8948 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "31d3ed0d91314eeca257815a1252a43a"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
mode: GRACEFUL
new_leader_uuid: "1d74da833ad34b2395cda075e2da0c04"
from {username='slave'} at 127.0.0.1:56290
I20260503 13:57:37.433548 8948 raft_consensus.cc:606] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Received request to transfer leadership to TS 1d74da833ad34b2395cda075e2da0c04
I20260503 13:57:37.437377 8948 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cd1db07b9e59487fac793c7ee6364ebc"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
mode: GRACEFUL
new_leader_uuid: "4f29308110ba4738b29e7adaab74f553"
from {username='slave'} at 127.0.0.1:56290
I20260503 13:57:37.437795 8948 raft_consensus.cc:606] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Received request to transfer leadership to TS 4f29308110ba4738b29e7adaab74f553
I20260503 13:57:37.452392 9143 consensus_queue.cc:1048] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.463014 9143 consensus_queue.cc:1048] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.463253 9168 consensus_queue.cc:1048] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.472618 9024 raft_consensus.cc:3060] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:57:37.474200 9143 raft_consensus.cc:993] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04: : Instructing follower aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:37.474617 9143 raft_consensus.cc:1081] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Signalling peer aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:37.476449 8948 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:56274
I20260503 13:57:37.477087 8948 raft_consensus.cc:493] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:37.478315 8948 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.480413 9187 consensus_queue.cc:1048] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.484097 8948 raft_consensus.cc:515] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.486048 8948 leader_election.cc:290] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 2 election: Requested vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.486977 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:37.487483 9098 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.489439 9187 raft_consensus.cc:993] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb: : Instructing follower 1d74da833ad34b2395cda075e2da0c04 to start an election
I20260503 13:57:37.489358 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:37.489871 9187 raft_consensus.cc:1081] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Signalling peer 1d74da833ad34b2395cda075e2da0c04 to start an election
I20260503 13:57:37.489933 9024 raft_consensus.cc:3055] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:37.491467 9023 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85"
dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
from {username='slave'} at 127.0.0.1:45790
I20260503 13:57:37.491231 9024 raft_consensus.cc:740] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
I20260503 13:57:37.491969 9023 raft_consensus.cc:493] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:37.492496 9023 raft_consensus.cc:3060] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.492954 9024 consensus_queue.cc:260] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.493652 9098 raft_consensus.cc:2468] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 2.
I20260503 13:57:37.494292 9024 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.494724 8914 leader_election.cc:304] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f29308110ba4738b29e7adaab74f553, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:37.495543 9187 raft_consensus.cc:2804] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:37.495987 9187 raft_consensus.cc:697] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 LEADER]: Becoming Leader. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:37.497889 9187 consensus_queue.cc:237] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.498982 9023 raft_consensus.cc:515] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.500252 9024 raft_consensus.cc:2468] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 2.
I20260503 13:57:37.500590 9023 leader_election.cc:290] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 2 election: Requested vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.501590 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:37.501929 8948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
I20260503 13:57:37.502373 9098 raft_consensus.cc:3060] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.502537 8948 raft_consensus.cc:3055] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:37.502950 8948 raft_consensus.cc:740] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:37.503541 8948 consensus_queue.cc:260] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.504511 8948 raft_consensus.cc:3060] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.505661 8847 catalog_manager.cc:5671] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb reported cstate change: term changed from 1 to 2, leader changed from 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194) to aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193). New cstate: current_term: 2 leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:37.508042 9098 raft_consensus.cc:2468] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 2.
I20260503 13:57:37.508885 8989 leader_election.cc:304] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters:
I20260503 13:57:37.508965 8948 raft_consensus.cc:2468] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 2.
I20260503 13:57:37.509511 9162 raft_consensus.cc:2804] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:37.509899 9162 raft_consensus.cc:697] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 2 LEADER]: Becoming Leader. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
I20260503 13:57:37.510648 9162 consensus_queue.cc:237] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.512750 9187 consensus_queue.cc:1048] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.517066 8847 catalog_manager.cc:5671] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 reported cstate change: term changed from 1 to 2, leader changed from aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193) to 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194). New cstate: current_term: 2 leader_uuid: "1d74da833ad34b2395cda075e2da0c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:37.524686 9187 consensus_queue.cc:1048] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.526796 9187 consensus_queue.cc:1048] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.535017 9168 raft_consensus.cc:993] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb: : Instructing follower 1d74da833ad34b2395cda075e2da0c04 to start an election
I20260503 13:57:37.535398 9168 raft_consensus.cc:1081] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Signalling peer 1d74da833ad34b2395cda075e2da0c04 to start an election
I20260503 13:57:37.537328 9022 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "31d3ed0d91314eeca257815a1252a43a"
dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
from {username='slave'} at 127.0.0.1:45790
I20260503 13:57:37.539479 9187 consensus_queue.cc:1048] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.540138 9022 raft_consensus.cc:493] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:37.540513 9022 raft_consensus.cc:3060] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.546360 9022 raft_consensus.cc:515] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.550457 9022 leader_election.cc:290] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 2 election: Requested vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.551965 8948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d3ed0d91314eeca257815a1252a43a" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
I20260503 13:57:37.552009 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d3ed0d91314eeca257815a1252a43a" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:37.552629 8948 raft_consensus.cc:3055] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:37.552971 8948 raft_consensus.cc:740] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:37.552981 9098 raft_consensus.cc:3060] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.553759 8948 consensus_queue.cc:260] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.554797 8948 raft_consensus.cc:3060] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.557614 9098 raft_consensus.cc:2468] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 2.
I20260503 13:57:37.558456 8989 leader_election.cc:304] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters:
I20260503 13:57:37.558862 8948 raft_consensus.cc:2468] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 2.
I20260503 13:57:37.559171 9162 raft_consensus.cc:2804] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:37.559607 9162 raft_consensus.cc:697] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [term 2 LEADER]: Becoming Leader. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
I20260503 13:57:37.560322 9162 consensus_queue.cc:237] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.565865 8846 catalog_manager.cc:5671] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 reported cstate change: term changed from 1 to 2, leader changed from aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193) to 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194). New cstate: current_term: 2 leader_uuid: "1d74da833ad34b2395cda075e2da0c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:37.711138 9146 consensus_queue.cc:1048] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.717438 9222 raft_consensus.cc:993] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553: : Instructing follower 1d74da833ad34b2395cda075e2da0c04 to start an election
I20260503 13:57:37.717882 9146 raft_consensus.cc:1081] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 1 LEADER]: Signalling peer 1d74da833ad34b2395cda075e2da0c04 to start an election
I20260503 13:57:37.719267 9024 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404"
dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
from {username='slave'} at 127.0.0.1:45804
I20260503 13:57:37.719767 9024 raft_consensus.cc:493] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:37.720134 9024 raft_consensus.cc:3060] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.722641 9222 consensus_queue.cc:1048] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.724231 9162 consensus_queue.cc:1048] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.728145 9024 raft_consensus.cc:515] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.732438 9024 leader_election.cc:290] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 2 election: Requested vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.734517 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:37.734679 8949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
I20260503 13:57:37.735069 9098 raft_consensus.cc:3055] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:37.735211 8949 raft_consensus.cc:3060] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.735401 9098 raft_consensus.cc:740] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Running, Role: LEADER
I20260503 13:57:37.736161 9098 consensus_queue.cc:260] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.736716 9158 raft_consensus.cc:993] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04: : Instructing follower aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:37.737088 9158 raft_consensus.cc:1081] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Signalling peer aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:37.737210 9098 raft_consensus.cc:3060] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.740119 8947 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fc3e5bb9588d4c5c9549b594d40df424"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:56274
I20260503 13:57:37.740659 8947 raft_consensus.cc:493] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:37.740988 8947 raft_consensus.cc:3060] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.741588 8949 raft_consensus.cc:2468] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 2.
I20260503 13:57:37.744825 9098 raft_consensus.cc:2468] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 2.
I20260503 13:57:37.745930 8988 leader_election.cc:304] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:37.746305 9162 consensus_queue.cc:1048] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.747030 9158 raft_consensus.cc:2804] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:37.747496 9158 raft_consensus.cc:697] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 2 LEADER]: Becoming Leader. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
I20260503 13:57:37.748287 9158 consensus_queue.cc:237] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.753350 8947 raft_consensus.cc:515] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.755209 8947 leader_election.cc:290] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 2 election: Requested vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.756115 9097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc3e5bb9588d4c5c9549b594d40df424" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:37.756721 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc3e5bb9588d4c5c9549b594d40df424" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:37.757226 9024 raft_consensus.cc:3055] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:37.757499 9024 raft_consensus.cc:740] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
I20260503 13:57:37.758188 9024 consensus_queue.cc:260] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.759168 9024 raft_consensus.cc:3060] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.761590 8847 catalog_manager.cc:5671] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 reported cstate change: term changed from 1 to 2, leader changed from 4f29308110ba4738b29e7adaab74f553 (127.8.26.195) to 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194). New cstate: current_term: 2 leader_uuid: "1d74da833ad34b2395cda075e2da0c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:37.764724 9024 raft_consensus.cc:2468] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 2.
I20260503 13:57:37.765527 8915 leader_election.cc:304] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb; no voters: 4f29308110ba4738b29e7adaab74f553
I20260503 13:57:37.766188 9168 raft_consensus.cc:2804] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:37.766610 9168 raft_consensus.cc:697] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 LEADER]: Becoming Leader. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:37.767268 9168 consensus_queue.cc:237] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.773973 8846 catalog_manager.cc:5671] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb reported cstate change: term changed from 1 to 2, leader changed from 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194) to aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193). New cstate: current_term: 2 leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:37.777423 9225 raft_consensus.cc:993] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04: : Instructing follower aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:37.777868 9143 raft_consensus.cc:1081] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Signalling peer aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:37.779191 8947 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:56274
I20260503 13:57:37.779574 8947 raft_consensus.cc:493] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:37.780236 8947 raft_consensus.cc:3060] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.785118 8947 raft_consensus.cc:515] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.786515 8947 leader_election.cc:290] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 2 election: Requested vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:37.787164 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:37.787412 9098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:37.787668 9024 raft_consensus.cc:3055] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:37.787849 9098 raft_consensus.cc:3060] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.787949 9024 raft_consensus.cc:740] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
I20260503 13:57:37.788689 9024 consensus_queue.cc:260] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.789512 9024 raft_consensus.cc:3060] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.792163 9098 raft_consensus.cc:2468] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 2.
I20260503 13:57:37.793030 8914 leader_election.cc:304] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f29308110ba4738b29e7adaab74f553, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:37.793602 9168 raft_consensus.cc:2804] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:37.793615 9024 raft_consensus.cc:2468] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 2.
I20260503 13:57:37.794071 9168 raft_consensus.cc:697] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 2 LEADER]: Becoming Leader. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:37.794766 9168 consensus_queue.cc:237] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.799988 8846 catalog_manager.cc:5671] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb reported cstate change: term changed from 1 to 2, leader changed from 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194) to aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193). New cstate: current_term: 2 leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:37.825865 9187 raft_consensus.cc:993] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb: : Instructing follower 4f29308110ba4738b29e7adaab74f553 to start an election
I20260503 13:57:37.826273 9187 raft_consensus.cc:1081] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Signalling peer 4f29308110ba4738b29e7adaab74f553 to start an election
I20260503 13:57:37.827659 9098 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cd1db07b9e59487fac793c7ee6364ebc"
dest_uuid: "4f29308110ba4738b29e7adaab74f553"
from {username='slave'} at 127.0.0.1:53720
I20260503 13:57:37.828459 9098 raft_consensus.cc:493] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:37.828835 9098 raft_consensus.cc:3060] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.833143 9098 raft_consensus.cc:515] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.834419 9098 leader_election.cc:290] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 2 election: Requested vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683)
I20260503 13:57:37.835141 8947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd1db07b9e59487fac793c7ee6364ebc" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
I20260503 13:57:37.835549 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd1db07b9e59487fac793c7ee6364ebc" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:37.835680 8947 raft_consensus.cc:3055] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:37.836040 8947 raft_consensus.cc:740] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:37.836061 9024 raft_consensus.cc:3060] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.836730 8947 consensus_queue.cc:260] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.837579 9145 consensus_queue.cc:1048] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.837846 8947 raft_consensus.cc:3060] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:37.839350 9145 consensus_queue.cc:1048] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:57:37.843133 9024 raft_consensus.cc:2468] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f29308110ba4738b29e7adaab74f553 in term 2.
I20260503 13:57:37.844115 8947 raft_consensus.cc:2468] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f29308110ba4738b29e7adaab74f553 in term 2.
I20260503 13:57:37.844352 9064 leader_election.cc:304] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters:
I20260503 13:57:37.845252 9222 raft_consensus.cc:2804] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:37.845686 9222 raft_consensus.cc:697] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [term 2 LEADER]: Becoming Leader. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Running, Role: LEADER
I20260503 13:57:37.846494 9222 consensus_queue.cc:237] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:37.861348 8845 catalog_manager.cc:5671] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 reported cstate change: term changed from 1 to 2, leader changed from aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193) to 4f29308110ba4738b29e7adaab74f553 (127.8.26.195). New cstate: current_term: 2 leader_uuid: "4f29308110ba4738b29e7adaab74f553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: HEALTHY } } }
I20260503 13:57:37.901446 9098 raft_consensus.cc:1275] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Refusing update from remote peer 1d74da833ad34b2395cda075e2da0c04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:37.902433 9143 consensus_queue.cc:1048] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:37.908494 8947 raft_consensus.cc:1275] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Refusing update from remote peer 1d74da833ad34b2395cda075e2da0c04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:37.909677 9225 consensus_queue.cc:1048] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:38.069450 9022 raft_consensus.cc:1275] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:38.070461 9187 consensus_queue.cc:1048] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:38.076946 9098 raft_consensus.cc:1275] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:38.078233 9145 consensus_queue.cc:1048] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:38.141249 8947 raft_consensus.cc:1275] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Refusing update from remote peer 1d74da833ad34b2395cda075e2da0c04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:38.142289 9225 consensus_queue.cc:1048] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:38.148698 9097 raft_consensus.cc:1275] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Refusing update from remote peer 1d74da833ad34b2395cda075e2da0c04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:38.150099 9225 consensus_queue.cc:1048] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:38.199747 8947 raft_consensus.cc:1275] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Refusing update from remote peer 1d74da833ad34b2395cda075e2da0c04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:38.200825 9225 consensus_queue.cc:1048] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:38.207427 9097 raft_consensus.cc:1275] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Refusing update from remote peer 1d74da833ad34b2395cda075e2da0c04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:38.208806 9225 consensus_queue.cc:1048] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:38.289693 9022 raft_consensus.cc:1275] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:38.290838 9168 consensus_queue.cc:1048] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:38.296442 9022 raft_consensus.cc:1275] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:38.298920 9168 consensus_queue.cc:1048] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:38.298761 9097 raft_consensus.cc:1275] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:38.301321 9168 consensus_queue.cc:1048] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:38.306288 9098 raft_consensus.cc:3060] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:38.312438 9098 raft_consensus.cc:1275] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:38.313956 9188 consensus_queue.cc:1048] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:38.358484 9024 raft_consensus.cc:1275] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Refusing update from remote peer 4f29308110ba4738b29e7adaab74f553: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:38.359543 9222 consensus_queue.cc:1048] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:38.369956 8947 raft_consensus.cc:1275] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Refusing update from remote peer 4f29308110ba4738b29e7adaab74f553: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:38.371145 9232 consensus_queue.cc:1048] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:40.464076 8299 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20260503 13:57:40.464691 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:40.478322 8947 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
mode: GRACEFUL
new_leader_uuid: "4f29308110ba4738b29e7adaab74f553"
from {username='slave'} at 127.0.0.1:37250
I20260503 13:57:40.478792 8947 raft_consensus.cc:606] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 LEADER]: Received request to transfer leadership to TS 4f29308110ba4738b29e7adaab74f553
I20260503 13:57:40.480786 8947 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
mode: GRACEFUL
new_leader_uuid: "4f29308110ba4738b29e7adaab74f553"
from {username='slave'} at 127.0.0.1:37250
I20260503 13:57:40.481204 8947 raft_consensus.cc:606] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 2 LEADER]: Received request to transfer leadership to TS 4f29308110ba4738b29e7adaab74f553
I20260503 13:57:40.482028 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260503 13:57:40.482409 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:40.564173 9240 raft_consensus.cc:993] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb: : Instructing follower 4f29308110ba4738b29e7adaab74f553 to start an election
I20260503 13:57:40.564618 9188 raft_consensus.cc:1081] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 2 LEADER]: Signalling peer 4f29308110ba4738b29e7adaab74f553 to start an election
I20260503 13:57:40.565918 9098 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c"
dest_uuid: "4f29308110ba4738b29e7adaab74f553"
from {username='slave'} at 127.0.0.1:53720
I20260503 13:57:40.566313 9098 raft_consensus.cc:493] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:40.566542 9098 raft_consensus.cc:3060] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:40.570468 9098 raft_consensus.cc:515] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:40.571650 9098 leader_election.cc:290] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 3 election: Requested vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683)
I20260503 13:57:40.572350 8947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
I20260503 13:57:40.572649 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:40.572858 8947 raft_consensus.cc:3055] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:57:40.573092 9024 raft_consensus.cc:3060] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:40.573155 8947 raft_consensus.cc:740] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:40.573834 8947 consensus_queue.cc:260] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:40.574687 8947 raft_consensus.cc:3060] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:40.576812 9024 raft_consensus.cc:2468] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f29308110ba4738b29e7adaab74f553 in term 3.
I20260503 13:57:40.577658 9064 leader_election.cc:304] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters:
I20260503 13:57:40.578402 9236 raft_consensus.cc:2804] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:57:40.579187 8947 raft_consensus.cc:2468] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f29308110ba4738b29e7adaab74f553 in term 3.
I20260503 13:57:40.579308 9236 raft_consensus.cc:697] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 3 LEADER]: Becoming Leader. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Running, Role: LEADER
I20260503 13:57:40.580112 9236 consensus_queue.cc:237] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:40.586352 8847 catalog_manager.cc:5671] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 reported cstate change: term changed from 2 to 3, leader changed from aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193) to 4f29308110ba4738b29e7adaab74f553 (127.8.26.195). New cstate: current_term: 3 leader_uuid: "4f29308110ba4738b29e7adaab74f553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: HEALTHY } } }
I20260503 13:57:40.704635 9240 raft_consensus.cc:993] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb: : Instructing follower 4f29308110ba4738b29e7adaab74f553 to start an election
I20260503 13:57:40.705044 9188 raft_consensus.cc:1081] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 LEADER]: Signalling peer 4f29308110ba4738b29e7adaab74f553 to start an election
I20260503 13:57:40.706272 9098 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871"
dest_uuid: "4f29308110ba4738b29e7adaab74f553"
from {username='slave'} at 127.0.0.1:53720
I20260503 13:57:40.706646 9098 raft_consensus.cc:493] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:40.706854 9098 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:40.710673 9098 raft_consensus.cc:515] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:40.711836 9098 leader_election.cc:290] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 3 election: Requested vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717)
I20260503 13:57:40.712481 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:40.712998 9024 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:40.712916 8947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
I20260503 13:57:40.713507 8947 raft_consensus.cc:3055] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:57:40.713737 8947 raft_consensus.cc:740] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:40.714382 8947 consensus_queue.cc:260] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:40.715245 8947 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:40.716792 9024 raft_consensus.cc:2468] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f29308110ba4738b29e7adaab74f553 in term 3.
I20260503 13:57:40.717558 9064 leader_election.cc:304] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters:
I20260503 13:57:40.718164 9236 raft_consensus.cc:2804] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:57:40.718665 9236 raft_consensus.cc:697] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 3 LEADER]: Becoming Leader. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Running, Role: LEADER
I20260503 13:57:40.719473 9236 consensus_queue.cc:237] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:40.720249 8947 raft_consensus.cc:2468] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f29308110ba4738b29e7adaab74f553 in term 3.
I20260503 13:57:40.726313 8847 catalog_manager.cc:5671] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 reported cstate change: term changed from 2 to 3, leader changed from aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193) to 4f29308110ba4738b29e7adaab74f553 (127.8.26.195). New cstate: current_term: 3 leader_uuid: "4f29308110ba4738b29e7adaab74f553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: HEALTHY } } }
I20260503 13:57:41.021119 8947 raft_consensus.cc:1275] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 3 FOLLOWER]: Refusing update from remote peer 4f29308110ba4738b29e7adaab74f553: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:41.022156 9236 consensus_queue.cc:1048] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:41.031092 9024 raft_consensus.cc:1275] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 3 FOLLOWER]: Refusing update from remote peer 4f29308110ba4738b29e7adaab74f553: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:41.039108 9241 consensus_queue.cc:1048] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260503 13:57:41.128517 9024 raft_consensus.cc:1275] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 3 FOLLOWER]: Refusing update from remote peer 4f29308110ba4738b29e7adaab74f553: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:41.129683 9251 consensus_queue.cc:1048] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:41.137284 8947 raft_consensus.cc:1275] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 FOLLOWER]: Refusing update from remote peer 4f29308110ba4738b29e7adaab74f553: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:41.138451 9251 consensus_queue.cc:1048] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:41.483268 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:41.488842 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:41.489243 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:42.490016 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:42.495221 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:42.495576 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:43.496358 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:43.501524 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:43.501896 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:44.502701 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:44.507833 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:44.508183 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:45.509280 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:45.516166 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:45.516712 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:46.517728 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:46.522714 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:46.523100 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:47.523950 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:47.528865 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:47.529251 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:48.529987 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:48.534869 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:48.535241 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:49.536010 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:49.541110 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:49.541450 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:50.566258 8299 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20260503 13:57:50.570077 9097 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871"
dest_uuid: "4f29308110ba4738b29e7adaab74f553"
mode: GRACEFUL
new_leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:53726
I20260503 13:57:50.570657 9097 raft_consensus.cc:606] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 3 LEADER]: Received request to transfer leadership to TS aea6a76deb4a4551af32917c5b8ff3eb
I20260503 13:57:50.576547 9097 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c"
dest_uuid: "4f29308110ba4738b29e7adaab74f553"
mode: GRACEFUL
new_leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:53726
I20260503 13:57:50.577097 9097 raft_consensus.cc:606] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 3 LEADER]: Received request to transfer leadership to TS aea6a76deb4a4551af32917c5b8ff3eb
I20260503 13:57:50.585418 9024 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85"
dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
mode: GRACEFUL
new_leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:45812
I20260503 13:57:50.585819 9024 raft_consensus.cc:606] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 2 LEADER]: Received request to transfer leadership to TS aea6a76deb4a4551af32917c5b8ff3eb
I20260503 13:57:50.589818 9024 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "65eae44a7ccd4e7eaf2b1aa1bb415221"
dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
mode: GRACEFUL
new_leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:45812
I20260503 13:57:50.590198 9024 raft_consensus.cc:606] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Received request to transfer leadership to TS aea6a76deb4a4551af32917c5b8ff3eb
I20260503 13:57:50.593753 9024 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404"
dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
mode: GRACEFUL
new_leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:45812
I20260503 13:57:50.594110 9024 raft_consensus.cc:606] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 2 LEADER]: Received request to transfer leadership to TS aea6a76deb4a4551af32917c5b8ff3eb
I20260503 13:57:50.779345 9256 raft_consensus.cc:993] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04: : Instructing follower aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:50.779865 9258 raft_consensus.cc:1081] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Signalling peer aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:50.781096 8949 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "65eae44a7ccd4e7eaf2b1aa1bb415221"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:56274
I20260503 13:57:50.781558 8949 raft_consensus.cc:493] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:50.781857 8949 raft_consensus.cc:3060] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:50.786963 8949 raft_consensus.cc:515] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:50.788645 8949 leader_election.cc:290] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 2 election: Requested vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:50.789258 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65eae44a7ccd4e7eaf2b1aa1bb415221" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:50.789719 9097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65eae44a7ccd4e7eaf2b1aa1bb415221" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:50.790007 9024 raft_consensus.cc:3055] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:50.790242 9097 raft_consensus.cc:3060] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:50.790287 9024 raft_consensus.cc:740] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
I20260503 13:57:50.791082 9024 consensus_queue.cc:260] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:50.792223 9024 raft_consensus.cc:3060] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:50.795300 9097 raft_consensus.cc:2468] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 2.
I20260503 13:57:50.796299 8914 leader_election.cc:304] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f29308110ba4738b29e7adaab74f553, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:50.797027 9188 raft_consensus.cc:2804] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:50.797848 9188 raft_consensus.cc:697] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 LEADER]: Becoming Leader. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:50.797807 9024 raft_consensus.cc:2468] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 2.
I20260503 13:57:50.798815 9188 consensus_queue.cc:237] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:50.805586 8846 catalog_manager.cc:5671] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb reported cstate change: term changed from 1 to 2, leader changed from 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194) to aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193). New cstate: current_term: 2 leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:50.908869 9236 raft_consensus.cc:993] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553: : Instructing follower aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:50.909260 9236 raft_consensus.cc:1081] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 3 LEADER]: Signalling peer aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:50.910549 8949 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:56280
I20260503 13:57:50.911015 8949 raft_consensus.cc:493] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:50.911266 8949 raft_consensus.cc:3060] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:50.915831 8949 raft_consensus.cc:515] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:50.917147 8949 leader_election.cc:290] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 4 election: Requested vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:50.917748 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:50.918179 9097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee4d02a278c0496db7cfef7d45a5f31c" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:50.918202 9024 raft_consensus.cc:3060] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:50.918587 9097 raft_consensus.cc:3055] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:57:50.918797 9097 raft_consensus.cc:740] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Running, Role: LEADER
I20260503 13:57:50.919287 9097 consensus_queue.cc:260] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:50.920099 9097 raft_consensus.cc:3060] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:50.922616 9024 raft_consensus.cc:2468] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 4.
I20260503 13:57:50.923818 8915 leader_election.cc:304] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:50.924417 9188 raft_consensus.cc:2804] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:57:50.924844 9188 raft_consensus.cc:697] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 4 LEADER]: Becoming Leader. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:50.925477 9097 raft_consensus.cc:2468] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 4.
I20260503 13:57:50.925572 9188 consensus_queue.cc:237] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:50.931833 8846 catalog_manager.cc:5671] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb reported cstate change: term changed from 3 to 4, leader changed from 4f29308110ba4738b29e7adaab74f553 (127.8.26.195) to aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193). New cstate: current_term: 4 leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:50.980371 9256 raft_consensus.cc:993] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04: : Instructing follower aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:50.980804 9258 raft_consensus.cc:1081] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 2 LEADER]: Signalling peer aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:50.981994 8949 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:56274
I20260503 13:57:50.982407 8949 raft_consensus.cc:493] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:50.982647 8949 raft_consensus.cc:3060] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:50.986793 8949 raft_consensus.cc:515] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:50.988124 8949 leader_election.cc:290] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 3 election: Requested vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:50.988894 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:50.989009 9097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:50.989390 9024 raft_consensus.cc:3055] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:57:50.989442 9097 raft_consensus.cc:3060] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:50.989673 9024 raft_consensus.cc:740] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
I20260503 13:57:50.990250 9024 consensus_queue.cc:260] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:50.991017 9024 raft_consensus.cc:3060] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:50.993216 9097 raft_consensus.cc:2468] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 3.
I20260503 13:57:50.993949 8914 leader_election.cc:304] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f29308110ba4738b29e7adaab74f553, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:50.994529 9188 raft_consensus.cc:2804] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:57:50.994830 9188 raft_consensus.cc:697] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 LEADER]: Becoming Leader. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:50.994877 9024 raft_consensus.cc:2468] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 3.
I20260503 13:57:50.995487 9188 consensus_queue.cc:237] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:51.001552 8846 catalog_manager.cc:5671] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb reported cstate change: term changed from 2 to 3, leader changed from 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194) to aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193). New cstate: current_term: 3 leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:51.041910 9265 raft_consensus.cc:993] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553: : Instructing follower aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:51.042331 9265 raft_consensus.cc:1081] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 3 LEADER]: Signalling peer aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:51.043650 8949 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:56280
I20260503 13:57:51.044085 8949 raft_consensus.cc:493] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:51.044363 8949 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:51.048609 8949 raft_consensus.cc:515] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:51.049943 8949 leader_election.cc:290] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 4 election: Requested vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:51.050626 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:51.050884 9097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:51.051079 9024 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:51.051327 9097 raft_consensus.cc:3055] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:57:51.051551 9097 raft_consensus.cc:740] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Running, Role: LEADER
I20260503 13:57:51.052241 9097 consensus_queue.cc:260] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:51.053074 9097 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:51.055263 9024 raft_consensus.cc:2468] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 4.
I20260503 13:57:51.056033 8915 leader_election.cc:304] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:51.056774 9188 raft_consensus.cc:2804] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:57:51.056823 9097 raft_consensus.cc:2468] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 4.
I20260503 13:57:51.057209 9188 raft_consensus.cc:697] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 LEADER]: Becoming Leader. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:51.057971 9188 consensus_queue.cc:237] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:51.063827 8846 catalog_manager.cc:5671] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb reported cstate change: term changed from 3 to 4, leader changed from 4f29308110ba4738b29e7adaab74f553 (127.8.26.195) to aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193). New cstate: current_term: 4 leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:51.109802 9256 raft_consensus.cc:993] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04: : Instructing follower aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:51.110175 9258 raft_consensus.cc:1081] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 2 LEADER]: Signalling peer aea6a76deb4a4551af32917c5b8ff3eb to start an election
I20260503 13:57:51.111397 8949 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
from {username='slave'} at 127.0.0.1:56274
I20260503 13:57:51.111810 8949 raft_consensus.cc:493] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:51.112066 8949 raft_consensus.cc:3060] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:51.116380 8949 raft_consensus.cc:515] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:51.117776 8949 leader_election.cc:290] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 3 election: Requested vote from peers 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:51.118456 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:51.118597 9097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85" candidate_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:51.118978 9024 raft_consensus.cc:3055] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:57:51.119012 9097 raft_consensus.cc:3060] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:51.119383 9024 raft_consensus.cc:740] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
I20260503 13:57:51.120062 9024 consensus_queue.cc:260] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:51.121193 9024 raft_consensus.cc:3060] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:57:51.124858 9097 raft_consensus.cc:2468] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 3.
I20260503 13:57:51.125726 8914 leader_election.cc:304] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f29308110ba4738b29e7adaab74f553, aea6a76deb4a4551af32917c5b8ff3eb; no voters:
I20260503 13:57:51.126312 9188 raft_consensus.cc:2804] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:57:51.126551 9024 raft_consensus.cc:2468] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea6a76deb4a4551af32917c5b8ff3eb in term 3.
I20260503 13:57:51.126628 9188 raft_consensus.cc:697] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 LEADER]: Becoming Leader. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:51.127374 9188 consensus_queue.cc:237] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:51.132771 8846 catalog_manager.cc:5671] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb reported cstate change: term changed from 2 to 3, leader changed from 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194) to aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193). New cstate: current_term: 3 leader_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:51.266204 9024 raft_consensus.cc:1275] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:51.267236 9188 consensus_queue.cc:1048] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:51.279320 9097 raft_consensus.cc:1275] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:51.280320 9188 consensus_queue.cc:1048] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:51.359548 9024 raft_consensus.cc:1275] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:57:51.360666 9188 consensus_queue.cc:1048] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:57:51.370257 9097 raft_consensus.cc:1275] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 4 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:57:51.371464 9188 consensus_queue.cc:1048] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:57:51.426180 9097 raft_consensus.cc:1275] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 3 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:51.427130 9188 consensus_queue.cc:1048] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:51.435078 9024 raft_consensus.cc:1275] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 3 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:51.436599 9277 consensus_queue.cc:1048] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260503 13:57:51.462883 9097 raft_consensus.cc:1275] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 4 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:57:51.463996 9277 consensus_queue.cc:1048] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:57:51.472196 9024 raft_consensus.cc:1275] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:57:51.473294 9277 consensus_queue.cc:1048] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:57:51.519739 9097 raft_consensus.cc:1275] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 3 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:51.520709 9277 consensus_queue.cc:1048] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:51.529384 9024 raft_consensus.cc:1275] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 3 FOLLOWER]: Refusing update from remote peer aea6a76deb4a4551af32917c5b8ff3eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:57:51.530525 9188 consensus_queue.cc:1048] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:57:53.623683 8299 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20260503 13:57:53.624233 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:53.631582 8949 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
mode: GRACEFUL
new_leader_uuid: "1d74da833ad34b2395cda075e2da0c04"
from {username='slave'} at 127.0.0.1:37250
I20260503 13:57:53.632031 8949 raft_consensus.cc:606] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 LEADER]: Received request to transfer leadership to TS 1d74da833ad34b2395cda075e2da0c04
I20260503 13:57:53.634102 8949 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
mode: GRACEFUL
new_leader_uuid: "4f29308110ba4738b29e7adaab74f553"
from {username='slave'} at 127.0.0.1:37250
I20260503 13:57:53.634555 8949 raft_consensus.cc:606] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 LEADER]: Received request to transfer leadership to TS 4f29308110ba4738b29e7adaab74f553
I20260503 13:57:53.636382 8949 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
mode: GRACEFUL
new_leader_uuid: "1d74da833ad34b2395cda075e2da0c04"
from {username='slave'} at 127.0.0.1:37250
I20260503 13:57:53.636826 8949 raft_consensus.cc:606] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 LEADER]: Received request to transfer leadership to TS 1d74da833ad34b2395cda075e2da0c04
I20260503 13:57:53.638535 8949 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e9d40d8eadbf4810ad68af958e0ffb30"
dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
mode: GRACEFUL
new_leader_uuid: "4f29308110ba4738b29e7adaab74f553"
from {username='slave'} at 127.0.0.1:37250
I20260503 13:57:53.638921 8949 raft_consensus.cc:606] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Received request to transfer leadership to TS 4f29308110ba4738b29e7adaab74f553
I20260503 13:57:53.639730 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260503 13:57:53.640093 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:53.898041 9188 raft_consensus.cc:993] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb: : Instructing follower 4f29308110ba4738b29e7adaab74f553 to start an election
I20260503 13:57:53.898466 9285 raft_consensus.cc:1081] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 LEADER]: Signalling peer 4f29308110ba4738b29e7adaab74f553 to start an election
I20260503 13:57:53.899649 9097 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85"
dest_uuid: "4f29308110ba4738b29e7adaab74f553"
from {username='slave'} at 127.0.0.1:53720
I20260503 13:57:53.900115 9097 raft_consensus.cc:493] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:53.900429 9097 raft_consensus.cc:3060] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:53.904533 9097 raft_consensus.cc:515] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:53.906106 9097 leader_election.cc:290] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 4 election: Requested vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683)
I20260503 13:57:53.906829 8949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
I20260503 13:57:53.907068 9022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "613bed3db7dd46aaa203a8d59930df85" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:53.907368 8949 raft_consensus.cc:3055] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:57:53.907518 9022 raft_consensus.cc:3060] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:53.907734 8949 raft_consensus.cc:740] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:53.908414 8949 consensus_queue.cc:260] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:53.909354 8949 raft_consensus.cc:3060] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:53.913305 9022 raft_consensus.cc:2468] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f29308110ba4738b29e7adaab74f553 in term 4.
I20260503 13:57:53.914285 9064 leader_election.cc:304] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters:
I20260503 13:57:53.914441 8949 raft_consensus.cc:2468] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f29308110ba4738b29e7adaab74f553 in term 4.
I20260503 13:57:53.914925 9265 raft_consensus.cc:2804] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:57:53.916141 9265 raft_consensus.cc:697] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 4 LEADER]: Becoming Leader. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Running, Role: LEADER
I20260503 13:57:53.916893 9265 consensus_queue.cc:237] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:53.922783 8846 catalog_manager.cc:5671] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 reported cstate change: term changed from 3 to 4, leader changed from aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193) to 4f29308110ba4738b29e7adaab74f553 (127.8.26.195). New cstate: current_term: 4 leader_uuid: "4f29308110ba4738b29e7adaab74f553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: HEALTHY } } }
I20260503 13:57:53.937915 9188 raft_consensus.cc:993] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb: : Instructing follower 1d74da833ad34b2395cda075e2da0c04 to start an election
I20260503 13:57:53.938228 9285 raft_consensus.cc:1081] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 LEADER]: Signalling peer 1d74da833ad34b2395cda075e2da0c04 to start an election
I20260503 13:57:53.939308 9024 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404"
dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
from {username='slave'} at 127.0.0.1:45790
I20260503 13:57:53.939675 9024 raft_consensus.cc:493] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:53.939926 9024 raft_consensus.cc:3060] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:53.943856 9024 raft_consensus.cc:515] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:53.945540 9024 leader_election.cc:290] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 4 election: Requested vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:53.946187 8949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
I20260503 13:57:53.946358 9097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df61c04d1414e0f9708d6fc99d92404" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:53.946651 8949 raft_consensus.cc:3055] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:57:53.946801 9097 raft_consensus.cc:3060] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:53.946941 8949 raft_consensus.cc:740] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:53.947551 8949 consensus_queue.cc:260] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:53.948256 8949 raft_consensus.cc:3060] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:57:53.950491 9097 raft_consensus.cc:2468] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 4.
I20260503 13:57:53.951370 8989 leader_election.cc:304] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters:
I20260503 13:57:53.951939 9258 raft_consensus.cc:2804] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:57:53.951997 8949 raft_consensus.cc:2468] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 4.
I20260503 13:57:53.952983 9258 raft_consensus.cc:697] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 4 LEADER]: Becoming Leader. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
I20260503 13:57:53.953644 9258 consensus_queue.cc:237] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:53.959306 8846 catalog_manager.cc:5671] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 reported cstate change: term changed from 3 to 4, leader changed from aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193) to 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194). New cstate: current_term: 4 leader_uuid: "1d74da833ad34b2395cda075e2da0c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:54.071839 9188 raft_consensus.cc:993] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb: : Instructing follower 4f29308110ba4738b29e7adaab74f553 to start an election
I20260503 13:57:54.072198 9285 raft_consensus.cc:1081] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Signalling peer 4f29308110ba4738b29e7adaab74f553 to start an election
I20260503 13:57:54.073340 9097 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e9d40d8eadbf4810ad68af958e0ffb30"
dest_uuid: "4f29308110ba4738b29e7adaab74f553"
from {username='slave'} at 127.0.0.1:53720
I20260503 13:57:54.073740 9097 raft_consensus.cc:493] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:54.073994 9097 raft_consensus.cc:3060] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:54.077881 9097 raft_consensus.cc:515] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:54.079188 9097 leader_election.cc:290] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 2 election: Requested vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683)
I20260503 13:57:54.079896 8949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9d40d8eadbf4810ad68af958e0ffb30" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
I20260503 13:57:54.079978 9024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9d40d8eadbf4810ad68af958e0ffb30" candidate_uuid: "4f29308110ba4738b29e7adaab74f553" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
I20260503 13:57:54.080298 8949 raft_consensus.cc:3055] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:57:54.080453 9024 raft_consensus.cc:3060] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:54.080652 8949 raft_consensus.cc:740] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:54.081279 8949 consensus_queue.cc:260] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:54.082085 8949 raft_consensus.cc:3060] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:57:54.085000 9024 raft_consensus.cc:2468] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f29308110ba4738b29e7adaab74f553 in term 2.
I20260503 13:57:54.085839 9064 leader_election.cc:304] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters:
I20260503 13:57:54.086406 9265 raft_consensus.cc:2804] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:57:54.086642 8949 raft_consensus.cc:2468] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f29308110ba4738b29e7adaab74f553 in term 2.
I20260503 13:57:54.086867 9265 raft_consensus.cc:697] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [term 2 LEADER]: Becoming Leader. State: Replica: 4f29308110ba4738b29e7adaab74f553, State: Running, Role: LEADER
I20260503 13:57:54.087651 9265 consensus_queue.cc:237] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:54.093870 8846 catalog_manager.cc:5671] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 reported cstate change: term changed from 1 to 2, leader changed from aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193) to 4f29308110ba4738b29e7adaab74f553 (127.8.26.195). New cstate: current_term: 2 leader_uuid: "4f29308110ba4738b29e7adaab74f553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: HEALTHY } } }
I20260503 13:57:54.150635 9285 raft_consensus.cc:993] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb: : Instructing follower 1d74da833ad34b2395cda075e2da0c04 to start an election
I20260503 13:57:54.151046 9295 raft_consensus.cc:1081] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 LEADER]: Signalling peer 1d74da833ad34b2395cda075e2da0c04 to start an election
I20260503 13:57:54.152376 9024 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871"
dest_uuid: "1d74da833ad34b2395cda075e2da0c04"
from {username='slave'} at 127.0.0.1:45790
I20260503 13:57:54.152889 9024 raft_consensus.cc:493] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260503 13:57:54.153244 9024 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:57:54.157768 9024 raft_consensus.cc:515] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:54.159005 9024 leader_election.cc:290] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 5 election: Requested vote from peers aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717), 4f29308110ba4738b29e7adaab74f553 (127.8.26.195:38053)
I20260503 13:57:54.159850 9097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "4f29308110ba4738b29e7adaab74f553"
I20260503 13:57:54.159950 8949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804ad412dec7415aaccb82e9bfa87871" candidate_uuid: "1d74da833ad34b2395cda075e2da0c04" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "aea6a76deb4a4551af32917c5b8ff3eb"
I20260503 13:57:54.160249 9097 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:57:54.160362 8949 raft_consensus.cc:3055] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:57:54.160672 8949 raft_consensus.cc:740] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: aea6a76deb4a4551af32917c5b8ff3eb, State: Running, Role: LEADER
I20260503 13:57:54.161196 8949 consensus_queue.cc:260] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:54.161936 8949 raft_consensus.cc:3060] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:57:54.164013 9097 raft_consensus.cc:2468] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 5.
I20260503 13:57:54.164784 8989 leader_election.cc:304] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d74da833ad34b2395cda075e2da0c04, 4f29308110ba4738b29e7adaab74f553; no voters:
I20260503 13:57:54.165343 9258 raft_consensus.cc:2804] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:57:54.165685 8949 raft_consensus.cc:2468] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d74da833ad34b2395cda075e2da0c04 in term 5.
I20260503 13:57:54.165699 9258 raft_consensus.cc:697] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 5 LEADER]: Becoming Leader. State: Replica: 1d74da833ad34b2395cda075e2da0c04, State: Running, Role: LEADER
I20260503 13:57:54.166608 9258 consensus_queue.cc:237] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } }
I20260503 13:57:54.172474 8846 catalog_manager.cc:5671] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 reported cstate change: term changed from 4 to 5, leader changed from aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193) to 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194). New cstate: current_term: 5 leader_uuid: "1d74da833ad34b2395cda075e2da0c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 } health_report { overall_health: UNKNOWN } } }
I20260503 13:57:54.315361 9024 raft_consensus.cc:1275] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Refusing update from remote peer 4f29308110ba4738b29e7adaab74f553: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:57:54.316269 9265 consensus_queue.cc:1048] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:57:54.326825 8949 raft_consensus.cc:1275] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 FOLLOWER]: Refusing update from remote peer 4f29308110ba4738b29e7adaab74f553: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:57:54.327850 9299 consensus_queue.cc:1048] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:57:54.427738 9097 raft_consensus.cc:1275] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 4 FOLLOWER]: Refusing update from remote peer 1d74da833ad34b2395cda075e2da0c04: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:57:54.428683 9258 consensus_queue.cc:1048] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:57:54.436442 8949 raft_consensus.cc:1275] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 FOLLOWER]: Refusing update from remote peer 1d74da833ad34b2395cda075e2da0c04: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:57:54.437810 9304 consensus_queue.cc:1048] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:57:54.482165 8949 raft_consensus.cc:1275] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Refusing update from remote peer 4f29308110ba4738b29e7adaab74f553: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:54.483100 9299 consensus_queue.cc:1048] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:54.490327 9024 raft_consensus.cc:1275] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Refusing update from remote peer 4f29308110ba4738b29e7adaab74f553: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:57:54.491724 9286 consensus_queue.cc:1048] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d74da833ad34b2395cda075e2da0c04" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:57:54.601399 8949 raft_consensus.cc:1275] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 5 FOLLOWER]: Refusing update from remote peer 1d74da833ad34b2395cda075e2da0c04: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260503 13:57:54.602339 9304 consensus_queue.cc:1048] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea6a76deb4a4551af32917c5b8ff3eb" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260503 13:57:54.609525 9097 raft_consensus.cc:1275] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 5 FOLLOWER]: Refusing update from remote peer 1d74da833ad34b2395cda075e2da0c04: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260503 13:57:54.610996 9292 consensus_queue.cc:1048] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f29308110ba4738b29e7adaab74f553" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260503 13:57:54.640923 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:54.645905 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:54.646255 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:55.647233 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:55.652902 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:55.653244 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:56.654117 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:56.659583 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:56.659951 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:57.660799 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:57.666143 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:57.666507 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:58.667260 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:57:58.672199 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:58.672609 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:57:59.884862 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260503 13:57:59.887512 8915 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:37683, user_credentials={real_user=slave}} blocked reactor thread for 212013us
I20260503 13:57:59.896246 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:57:59.896987 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:58:00.897991 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:58:00.903507 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:58:00.903867 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260503 13:58:00.907188 9045 debug-util.cc:398] Leaking SignalData structure 0x7b08001048c0 after lost signal to thread 8302
W20260503 13:58:00.907881 9045 debug-util.cc:398] Leaking SignalData structure 0x7b0800104fa0 after lost signal to thread 8881
W20260503 13:58:00.908648 9045 debug-util.cc:398] Leaking SignalData structure 0x7b08000faca0 after lost signal to thread 8973
W20260503 13:58:00.909293 9045 debug-util.cc:398] Leaking SignalData structure 0x7b0800178140 after lost signal to thread 9048
W20260503 13:58:00.909947 9045 debug-util.cc:398] Leaking SignalData structure 0x7b08000591a0 after lost signal to thread 9122
I20260503 13:58:01.904742 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:58:01.909981 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:58:01.910368 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:58:02.911134 8299 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260503 13:58:02.917250 8299 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260503 13:58:02.917604 8299 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260503 13:58:03.944166 8299 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20260503 13:58:03.944669 8299 tablet_server.cc:179] TabletServer@127.8.26.193:0 shutting down...
I20260503 13:58:03.964717 8299 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260503 13:58:03.965292 8299 tablet_replica.cc:333] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb: stopping tablet replica
I20260503 13:58:03.966086 8299 raft_consensus.cc:2243] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 4 LEADER]: Raft consensus shutting down.
I20260503 13:58:03.967252 8299 raft_consensus.cc:2272] T ee4d02a278c0496db7cfef7d45a5f31c P aea6a76deb4a4551af32917c5b8ff3eb [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:03.969310 8299 tablet_replica.cc:333] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb: stopping tablet replica
I20260503 13:58:03.969744 8299 raft_consensus.cc:2243] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:03.970120 8299 raft_consensus.cc:2272] T e9d40d8eadbf4810ad68af958e0ffb30 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:03.971629 8299 tablet_replica.cc:333] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb: stopping tablet replica
I20260503 13:58:03.972046 8299 raft_consensus.cc:2243] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:03.972388 8299 raft_consensus.cc:2272] T 5df61c04d1414e0f9708d6fc99d92404 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:03.973945 8299 tablet_replica.cc:333] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb: stopping tablet replica
I20260503 13:58:03.974399 8299 raft_consensus.cc:2243] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 LEADER]: Raft consensus shutting down.
I20260503 13:58:03.975063 8299 raft_consensus.cc:2272] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P aea6a76deb4a4551af32917c5b8ff3eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:03.976671 8299 tablet_replica.cc:333] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb: stopping tablet replica
I20260503 13:58:03.977104 8299 raft_consensus.cc:2243] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 LEADER]: Raft consensus shutting down.
I20260503 13:58:03.977782 8299 raft_consensus.cc:2272] T fc3e5bb9588d4c5c9549b594d40df424 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:03.979296 8299 tablet_replica.cc:333] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb: stopping tablet replica
I20260503 13:58:03.979754 8299 raft_consensus.cc:2243] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 LEADER]: Raft consensus shutting down.
I20260503 13:58:03.980674 8299 raft_consensus.cc:2272] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Raft consensus is shut down!
W20260503 13:58:03.982627 9062 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
I20260503 13:58:03.982844 8299 tablet_replica.cc:333] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb: stopping tablet replica
I20260503 13:58:03.983353 8299 raft_consensus.cc:2243] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 5 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:03.983834 8299 raft_consensus.cc:2272] T 804ad412dec7415aaccb82e9bfa87871 P aea6a76deb4a4551af32917c5b8ff3eb [term 5 FOLLOWER]: Raft consensus is shut down!
W20260503 13:58:03.985505 9062 consensus_peers.cc:597] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 -> Peer aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717): Couldn't send request to peer aea6a76deb4a4551af32917c5b8ff3eb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260503 13:58:03.985921 8299 tablet_replica.cc:333] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb: stopping tablet replica
I20260503 13:58:03.986375 8299 raft_consensus.cc:2243] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:03.986764 8299 raft_consensus.cc:2272] T 31d3ed0d91314eeca257815a1252a43a P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:03.988344 8299 tablet_replica.cc:333] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb: stopping tablet replica
I20260503 13:58:03.988813 8299 raft_consensus.cc:2243] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:03.989173 8299 raft_consensus.cc:2272] T cd1db07b9e59487fac793c7ee6364ebc P aea6a76deb4a4551af32917c5b8ff3eb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:03.990648 8299 tablet_replica.cc:333] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb: stopping tablet replica
I20260503 13:58:03.991067 8299 raft_consensus.cc:2243] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:03.991426 8299 raft_consensus.cc:2272] T 613bed3db7dd46aaa203a8d59930df85 P aea6a76deb4a4551af32917c5b8ff3eb [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.019035 8299 tablet_server.cc:196] TabletServer@127.8.26.193:0 shutdown complete.
I20260503 13:58:04.037592 8299 tablet_server.cc:179] TabletServer@127.8.26.194:0 shutting down...
W20260503 13:58:04.043676 9062 consensus_peers.cc:597] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 -> Peer aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717): Couldn't send request to peer aea6a76deb4a4551af32917c5b8ff3eb. Status: Network error: Client connection negotiation failed: client connection to 127.8.26.193:39717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260503 13:58:04.060464 8299 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260503 13:58:04.061094 8299 tablet_replica.cc:333] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04: stopping tablet replica
I20260503 13:58:04.061730 8299 raft_consensus.cc:2243] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:04.062230 8299 raft_consensus.cc:2272] T 613bed3db7dd46aaa203a8d59930df85 P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.063977 8299 tablet_replica.cc:333] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04: stopping tablet replica
I20260503 13:58:04.064441 8299 raft_consensus.cc:2243] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:04.064920 8299 raft_consensus.cc:2272] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 1d74da833ad34b2395cda075e2da0c04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.066473 8299 tablet_replica.cc:333] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04: stopping tablet replica
I20260503 13:58:04.066910 8299 raft_consensus.cc:2243] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [term 2 LEADER]: Raft consensus shutting down.
I20260503 13:58:04.067619 8299 raft_consensus.cc:2272] T 31d3ed0d91314eeca257815a1252a43a P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.069105 8299 tablet_replica.cc:333] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04: stopping tablet replica
I20260503 13:58:04.069530 8299 raft_consensus.cc:2243] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 5 LEADER]: Raft consensus shutting down.
I20260503 13:58:04.070196 8299 raft_consensus.cc:2272] T 804ad412dec7415aaccb82e9bfa87871 P 1d74da833ad34b2395cda075e2da0c04 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.072216 8299 tablet_replica.cc:333] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04: stopping tablet replica
I20260503 13:58:04.072746 8299 raft_consensus.cc:2243] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:04.073154 8299 raft_consensus.cc:2272] T cd1db07b9e59487fac793c7ee6364ebc P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.074946 8299 tablet_replica.cc:333] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04: stopping tablet replica
I20260503 13:58:04.075379 8299 raft_consensus.cc:2243] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:04.075760 8299 raft_consensus.cc:2272] T fc3e5bb9588d4c5c9549b594d40df424 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260503 13:58:04.076232 8988 consensus_peers.cc:597] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 -> Peer aea6a76deb4a4551af32917c5b8ff3eb (127.8.26.193:39717): Couldn't send request to peer aea6a76deb4a4551af32917c5b8ff3eb. Status: Network error: Client connection negotiation failed: client connection to 127.8.26.193:39717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260503 13:58:04.077626 8299 tablet_replica.cc:333] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04: stopping tablet replica
I20260503 13:58:04.078061 8299 raft_consensus.cc:2243] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:04.078390 8299 raft_consensus.cc:2272] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.079982 8299 tablet_replica.cc:333] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04: stopping tablet replica
I20260503 13:58:04.080376 8299 raft_consensus.cc:2243] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:04.080730 8299 raft_consensus.cc:2272] T ee4d02a278c0496db7cfef7d45a5f31c P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.082208 8299 tablet_replica.cc:333] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04: stopping tablet replica
I20260503 13:58:04.082590 8299 raft_consensus.cc:2243] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 4 LEADER]: Raft consensus shutting down.
I20260503 13:58:04.083222 8299 raft_consensus.cc:2272] T 5df61c04d1414e0f9708d6fc99d92404 P 1d74da833ad34b2395cda075e2da0c04 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.084848 8299 tablet_replica.cc:333] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04: stopping tablet replica
I20260503 13:58:04.085217 8299 raft_consensus.cc:2243] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:04.085541 8299 raft_consensus.cc:2272] T e9d40d8eadbf4810ad68af958e0ffb30 P 1d74da833ad34b2395cda075e2da0c04 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260503 13:58:04.112924 9064 consensus_peers.cc:597] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 -> Peer 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683): Couldn't send request to peer 1d74da833ad34b2395cda075e2da0c04. Status: Network error: Client connection negotiation failed: client connection to 127.8.26.194:37683: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260503 13:58:04.135562 8299 tablet_server.cc:196] TabletServer@127.8.26.194:0 shutdown complete.
W20260503 13:58:04.146063 9064 consensus_peers.cc:597] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 -> Peer 1d74da833ad34b2395cda075e2da0c04 (127.8.26.194:37683): Couldn't send request to peer 1d74da833ad34b2395cda075e2da0c04. Status: Network error: Client connection negotiation failed: client connection to 127.8.26.194:37683: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260503 13:58:04.157488 8299 tablet_server.cc:179] TabletServer@127.8.26.195:0 shutting down...
I20260503 13:58:04.179296 8299 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260503 13:58:04.179939 8299 tablet_replica.cc:333] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553: stopping tablet replica
I20260503 13:58:04.180497 8299 raft_consensus.cc:2243] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:04.181010 8299 raft_consensus.cc:2272] T 5df61c04d1414e0f9708d6fc99d92404 P 4f29308110ba4738b29e7adaab74f553 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.182765 8299 tablet_replica.cc:333] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553: stopping tablet replica
I20260503 13:58:04.183218 8299 raft_consensus.cc:2243] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [term 2 LEADER]: Raft consensus shutting down.
I20260503 13:58:04.183902 8299 raft_consensus.cc:2272] T e9d40d8eadbf4810ad68af958e0ffb30 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.185546 8299 tablet_replica.cc:333] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553: stopping tablet replica
I20260503 13:58:04.186007 8299 raft_consensus.cc:2243] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:04.186401 8299 raft_consensus.cc:2272] T 31d3ed0d91314eeca257815a1252a43a P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.188125 8299 tablet_replica.cc:333] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553: stopping tablet replica
I20260503 13:58:04.188630 8299 raft_consensus.cc:2243] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:04.189029 8299 raft_consensus.cc:2272] T 804ad412dec7415aaccb82e9bfa87871 P 4f29308110ba4738b29e7adaab74f553 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.190513 8299 tablet_replica.cc:333] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553: stopping tablet replica
I20260503 13:58:04.190948 8299 raft_consensus.cc:2243] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 4 LEADER]: Raft consensus shutting down.
I20260503 13:58:04.191634 8299 raft_consensus.cc:2272] T 613bed3db7dd46aaa203a8d59930df85 P 4f29308110ba4738b29e7adaab74f553 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.193217 8299 tablet_replica.cc:333] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553: stopping tablet replica
I20260503 13:58:04.193619 8299 raft_consensus.cc:2243] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [term 2 LEADER]: Raft consensus shutting down.
I20260503 13:58:04.194272 8299 raft_consensus.cc:2272] T cd1db07b9e59487fac793c7ee6364ebc P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.195817 8299 tablet_replica.cc:333] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553: stopping tablet replica
I20260503 13:58:04.196218 8299 raft_consensus.cc:2243] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:04.196530 8299 raft_consensus.cc:2272] T 8f810c64c8f24fe8a8b85a155f4ce8e4 P 4f29308110ba4738b29e7adaab74f553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.198098 8299 tablet_replica.cc:333] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553: stopping tablet replica
I20260503 13:58:04.198519 8299 raft_consensus.cc:2243] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:04.198877 8299 raft_consensus.cc:2272] T 65eae44a7ccd4e7eaf2b1aa1bb415221 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.200435 8299 tablet_replica.cc:333] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553: stopping tablet replica
I20260503 13:58:04.200866 8299 raft_consensus.cc:2243] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:04.201193 8299 raft_consensus.cc:2272] T fc3e5bb9588d4c5c9549b594d40df424 P 4f29308110ba4738b29e7adaab74f553 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.202741 8299 tablet_replica.cc:333] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553: stopping tablet replica
I20260503 13:58:04.203120 8299 raft_consensus.cc:2243] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260503 13:58:04.203482 8299 raft_consensus.cc:2272] T ee4d02a278c0496db7cfef7d45a5f31c P 4f29308110ba4738b29e7adaab74f553 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.230727 8299 tablet_server.cc:196] TabletServer@127.8.26.195:0 shutdown complete.
I20260503 13:58:04.248625 8299 master.cc:562] Master@127.8.26.254:39623 shutting down...
I20260503 13:58:04.266345 8299 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [term 1 LEADER]: Raft consensus shutting down.
I20260503 13:58:04.266937 8299 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260503 13:58:04.267274 8299 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2fa153c1e2264aa2ae907ea5273ca055: stopping tablet replica
I20260503 13:58:04.285723 8299 master.cc:584] Master@127.8.26.254:39623 shutdown complete.
1d74da833ad34b2395cda075e2da0c04 4
4f29308110ba4738b29e7adaab74f553 1
aea6a76deb4a4551af32917c5b8ff3eb 5
1d74da833ad34b2395cda075e2da0c04 4
4f29308110ba4738b29e7adaab74f553 3
aea6a76deb4a4551af32917c5b8ff3eb 3
1d74da833ad34b2395cda075e2da0c04 1
4f29308110ba4738b29e7adaab74f553 1
aea6a76deb4a4551af32917c5b8ff3eb 8
1d74da833ad34b2395cda075e2da0c04 3
4f29308110ba4738b29e7adaab74f553 3
aea6a76deb4a4551af32917c5b8ff3eb 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28649 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260503 13:58:04.325227 8299 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.8.26.254:40413
I20260503 13:58:04.326272 8299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260503 13:58:04.331409 9327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:58:04.331439 9326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:58:04.333981 9329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260503 13:58:04.334188 8299 server_base.cc:1061] running on GCE node
I20260503 13:58:04.335108 8299 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260503 13:58:04.335290 8299 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260503 13:58:04.335425 8299 hybrid_clock.cc:648] HybridClock initialized: now 1777816684335408 us; error 0 us; skew 500 ppm
I20260503 13:58:04.335891 8299 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260503 13:58:04.338212 8299 webserver.cc:492] Webserver started at http://127.8.26.254:36073/ using document root <none> and password file <none>
I20260503 13:58:04.338649 8299 fs_manager.cc:362] Metadata directory not provided
I20260503 13:58:04.338821 8299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260503 13:58:04.339062 8299 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260503 13:58:04.340129 8299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/master-0-root/instance:
uuid: "23e911af899f462ba467ed1dfe5a5f91"
format_stamp: "Formatted at 2026-05-03 13:58:04 on dist-test-slave-jcqq"
I20260503 13:58:04.344205 8299 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260503 13:58:04.347437 9334 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260503 13:58:04.348125 8299 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260503 13:58:04.348387 8299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/master-0-root
uuid: "23e911af899f462ba467ed1dfe5a5f91"
format_stamp: "Formatted at 2026-05-03 13:58:04 on dist-test-slave-jcqq"
I20260503 13:58:04.348659 8299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260503 13:58:04.379149 8299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260503 13:58:04.380347 8299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260503 13:58:04.415010 8299 rpc_server.cc:307] RPC server started. Bound to: 127.8.26.254:40413
I20260503 13:58:04.415092 9385 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.26.254:40413 every 8 connection(s)
I20260503 13:58:04.418591 9386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:04.427759 9386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Bootstrap starting.
I20260503 13:58:04.432106 9386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:04.435941 9386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: No bootstrap required, opened a new log
I20260503 13:58:04.438128 9386 raft_consensus.cc:359] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e911af899f462ba467ed1dfe5a5f91" member_type: VOTER }
I20260503 13:58:04.438527 9386 raft_consensus.cc:385] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:04.438710 9386 raft_consensus.cc:740] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23e911af899f462ba467ed1dfe5a5f91, State: Initialized, Role: FOLLOWER
I20260503 13:58:04.439131 9386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e911af899f462ba467ed1dfe5a5f91" member_type: VOTER }
I20260503 13:58:04.439534 9386 raft_consensus.cc:399] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260503 13:58:04.439733 9386 raft_consensus.cc:493] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260503 13:58:04.439925 9386 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:04.443775 9386 raft_consensus.cc:515] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e911af899f462ba467ed1dfe5a5f91" member_type: VOTER }
I20260503 13:58:04.444202 9386 leader_election.cc:304] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23e911af899f462ba467ed1dfe5a5f91; no voters:
I20260503 13:58:04.445225 9386 leader_election.cc:290] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260503 13:58:04.445590 9389 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:04.446848 9389 raft_consensus.cc:697] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [term 1 LEADER]: Becoming Leader. State: Replica: 23e911af899f462ba467ed1dfe5a5f91, State: Running, Role: LEADER
I20260503 13:58:04.447402 9389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e911af899f462ba467ed1dfe5a5f91" member_type: VOTER }
I20260503 13:58:04.447768 9386 sys_catalog.cc:565] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [sys.catalog]: configured and running, proceeding with master startup.
I20260503 13:58:04.451918 9391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23e911af899f462ba467ed1dfe5a5f91. Latest consensus state: current_term: 1 leader_uuid: "23e911af899f462ba467ed1dfe5a5f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e911af899f462ba467ed1dfe5a5f91" member_type: VOTER } }
I20260503 13:58:04.452436 9391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [sys.catalog]: This master's current role is: LEADER
I20260503 13:58:04.453248 9390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23e911af899f462ba467ed1dfe5a5f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e911af899f462ba467ed1dfe5a5f91" member_type: VOTER } }
I20260503 13:58:04.453733 9390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91 [sys.catalog]: This master's current role is: LEADER
I20260503 13:58:04.454330 9397 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260503 13:58:04.458865 9397 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260503 13:58:04.461998 8299 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260503 13:58:04.466928 9397 catalog_manager.cc:1357] Generated new cluster ID: c8da495c5bc343beb00049714232a459
I20260503 13:58:04.467180 9397 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260503 13:58:04.477272 9397 catalog_manager.cc:1380] Generated new certificate authority record
I20260503 13:58:04.478302 9397 catalog_manager.cc:1514] Loading token signing keys...
I20260503 13:58:04.489297 9397 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Generated new TSK 0
I20260503 13:58:04.489795 9397 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260503 13:58:04.495867 8299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260503 13:58:04.501247 9407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:58:04.502302 9408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:58:04.503523 9410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260503 13:58:04.504290 8299 server_base.cc:1061] running on GCE node
I20260503 13:58:04.505089 8299 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260503 13:58:04.505266 8299 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260503 13:58:04.505414 8299 hybrid_clock.cc:648] HybridClock initialized: now 1777816684505397 us; error 0 us; skew 500 ppm
I20260503 13:58:04.505903 8299 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260503 13:58:04.507898 8299 webserver.cc:492] Webserver started at http://127.8.26.193:41259/ using document root <none> and password file <none>
I20260503 13:58:04.508328 8299 fs_manager.cc:362] Metadata directory not provided
I20260503 13:58:04.508494 8299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260503 13:58:04.508785 8299 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260503 13:58:04.509748 8299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-0-root/instance:
uuid: "1cb63886040047559d3893923cc9aaf6"
format_stamp: "Formatted at 2026-05-03 13:58:04 on dist-test-slave-jcqq"
I20260503 13:58:04.513700 8299 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260503 13:58:04.516474 9415 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260503 13:58:04.517159 8299 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260503 13:58:04.517405 8299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-0-root
uuid: "1cb63886040047559d3893923cc9aaf6"
format_stamp: "Formatted at 2026-05-03 13:58:04 on dist-test-slave-jcqq"
I20260503 13:58:04.517653 8299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260503 13:58:04.531304 8299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260503 13:58:04.532439 8299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260503 13:58:04.533813 8299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260503 13:58:04.535880 8299 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260503 13:58:04.536065 8299 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260503 13:58:04.536273 8299 ts_tablet_manager.cc:616] Registered 0 tablets
I20260503 13:58:04.536428 8299 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260503 13:58:04.574867 8299 rpc_server.cc:307] RPC server started. Bound to: 127.8.26.193:45599
I20260503 13:58:04.574959 9477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.26.193:45599 every 8 connection(s)
I20260503 13:58:04.579011 8299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260503 13:58:04.585892 9482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:58:04.587411 9483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:58:04.588986 9485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260503 13:58:04.589598 8299 server_base.cc:1061] running on GCE node
I20260503 13:58:04.590750 8299 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260503 13:58:04.590927 8299 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260503 13:58:04.591048 8299 hybrid_clock.cc:648] HybridClock initialized: now 1777816684591037 us; error 0 us; skew 500 ppm
I20260503 13:58:04.591540 8299 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260503 13:58:04.593876 8299 webserver.cc:492] Webserver started at http://127.8.26.194:44863/ using document root <none> and password file <none>
I20260503 13:58:04.594383 8299 fs_manager.cc:362] Metadata directory not provided
I20260503 13:58:04.594621 8299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260503 13:58:04.594620 9478 heartbeater.cc:344] Connected to a master server at 127.8.26.254:40413
I20260503 13:58:04.594985 8299 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260503 13:58:04.595044 9478 heartbeater.cc:461] Registering TS with master...
I20260503 13:58:04.595755 9478 heartbeater.cc:507] Master 127.8.26.254:40413 requested a full tablet report, sending...
I20260503 13:58:04.596083 8299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-1-root/instance:
uuid: "428b342dc4f041df956015e7d7cd173a"
format_stamp: "Formatted at 2026-05-03 13:58:04 on dist-test-slave-jcqq"
I20260503 13:58:04.597571 9351 ts_manager.cc:194] Registered new tserver with Master: 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:04.599131 9351 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51388
I20260503 13:58:04.600486 8299 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260503 13:58:04.603451 9490 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260503 13:58:04.604113 8299 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260503 13:58:04.604352 8299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-1-root
uuid: "428b342dc4f041df956015e7d7cd173a"
format_stamp: "Formatted at 2026-05-03 13:58:04 on dist-test-slave-jcqq"
I20260503 13:58:04.604635 8299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260503 13:58:04.633749 8299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260503 13:58:04.634932 8299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260503 13:58:04.636193 8299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260503 13:58:04.638306 8299 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260503 13:58:04.638477 8299 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260503 13:58:04.638703 8299 ts_tablet_manager.cc:616] Registered 0 tablets
I20260503 13:58:04.638839 8299 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260503 13:58:04.674821 8299 rpc_server.cc:307] RPC server started. Bound to: 127.8.26.194:46805
I20260503 13:58:04.674932 9552 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.26.194:46805 every 8 connection(s)
I20260503 13:58:04.678941 8299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260503 13:58:04.685055 9556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:58:04.686854 9557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260503 13:58:04.688130 9559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260503 13:58:04.688550 8299 server_base.cc:1061] running on GCE node
I20260503 13:58:04.689559 8299 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260503 13:58:04.689743 8299 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260503 13:58:04.689905 8299 hybrid_clock.cc:648] HybridClock initialized: now 1777816684689889 us; error 0 us; skew 500 ppm
I20260503 13:58:04.690111 9553 heartbeater.cc:344] Connected to a master server at 127.8.26.254:40413
I20260503 13:58:04.690465 9553 heartbeater.cc:461] Registering TS with master...
I20260503 13:58:04.690495 8299 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260503 13:58:04.691344 9553 heartbeater.cc:507] Master 127.8.26.254:40413 requested a full tablet report, sending...
I20260503 13:58:04.693179 8299 webserver.cc:492] Webserver started at http://127.8.26.195:35921/ using document root <none> and password file <none>
I20260503 13:58:04.693501 9351 ts_manager.cc:194] Registered new tserver with Master: 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:04.693779 8299 fs_manager.cc:362] Metadata directory not provided
I20260503 13:58:04.694028 8299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260503 13:58:04.694310 8299 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260503 13:58:04.695183 9351 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51400
I20260503 13:58:04.695698 8299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-2-root/instance:
uuid: "4bea375a55034750a1f8804b0db97868"
format_stamp: "Formatted at 2026-05-03 13:58:04 on dist-test-slave-jcqq"
I20260503 13:58:04.700086 8299 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260503 13:58:04.703286 9564 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260503 13:58:04.704001 8299 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260503 13:58:04.704260 8299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-2-root
uuid: "4bea375a55034750a1f8804b0db97868"
format_stamp: "Formatted at 2026-05-03 13:58:04 on dist-test-slave-jcqq"
I20260503 13:58:04.704504 8299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwffn2h/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777816625017654-8299-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260503 13:58:04.723264 8299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260503 13:58:04.724355 8299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260503 13:58:04.725664 8299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260503 13:58:04.727630 8299 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260503 13:58:04.727813 8299 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260503 13:58:04.728024 8299 ts_tablet_manager.cc:616] Registered 0 tablets
I20260503 13:58:04.728171 8299 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260503 13:58:04.764014 8299 rpc_server.cc:307] RPC server started. Bound to: 127.8.26.195:39997
I20260503 13:58:04.764103 9626 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.26.195:39997 every 8 connection(s)
I20260503 13:58:04.776352 9627 heartbeater.cc:344] Connected to a master server at 127.8.26.254:40413
I20260503 13:58:04.776710 9627 heartbeater.cc:461] Registering TS with master...
I20260503 13:58:04.777361 9627 heartbeater.cc:507] Master 127.8.26.254:40413 requested a full tablet report, sending...
I20260503 13:58:04.779039 9351 ts_manager.cc:194] Registered new tserver with Master: 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:04.780138 8299 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01329222s
I20260503 13:58:04.780357 9351 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51410
I20260503 13:58:04.802899 9351 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51412:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260503 13:58:04.808936 9351 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260503 13:58:05.020180 9518 tablet_service.cc:1511] Processing CreateTablet for tablet 32fd983d73354e649ff91e64f251e44f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260503 13:58:05.020469 9517 tablet_service.cc:1511] Processing CreateTablet for tablet 497cd68663ac4d34b95230215aa13bd2 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260503 13:58:05.021647 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32fd983d73354e649ff91e64f251e44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.022176 9513 tablet_service.cc:1511] Processing CreateTablet for tablet 83ebc0b241994520929db0a8b7fb8546 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260503 13:58:05.022863 9591 tablet_service.cc:1511] Processing CreateTablet for tablet 497cd68663ac4d34b95230215aa13bd2 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260503 13:58:05.024092 9591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 497cd68663ac4d34b95230215aa13bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.025066 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 497cd68663ac4d34b95230215aa13bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.025197 9516 tablet_service.cc:1511] Processing CreateTablet for tablet ec9b187575714b4884a75ff6aa911e15 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260503 13:58:05.026368 9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec9b187575714b4884a75ff6aa911e15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.028985 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83ebc0b241994520929db0a8b7fb8546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.029628 9590 tablet_service.cc:1511] Processing CreateTablet for tablet ec9b187575714b4884a75ff6aa911e15 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260503 13:58:05.030761 9590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec9b187575714b4884a75ff6aa911e15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.021729 9592 tablet_service.cc:1511] Processing CreateTablet for tablet 32fd983d73354e649ff91e64f251e44f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260503 13:58:05.033830 9592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32fd983d73354e649ff91e64f251e44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.033630 9515 tablet_service.cc:1511] Processing CreateTablet for tablet b3b7ab9cca2847d392329f5c80956bbd (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260503 13:58:05.034468 9510 tablet_service.cc:1511] Processing CreateTablet for tablet ddaccadb96de46d3a03ca8d2f3a8c6e0 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260503 13:58:05.034888 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b7ab9cca2847d392329f5c80956bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.035669 9510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddaccadb96de46d3a03ca8d2f3a8c6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.035773 9512 tablet_service.cc:1511] Processing CreateTablet for tablet 57b1804f75444731b8b3b6af3c038cf6 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260503 13:58:05.037048 9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57b1804f75444731b8b3b6af3c038cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.024010 9514 tablet_service.cc:1511] Processing CreateTablet for tablet cefaab375804454f943c82f15b8fd68b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260503 13:58:05.039433 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cefaab375804454f943c82f15b8fd68b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.039142 9511 tablet_service.cc:1511] Processing CreateTablet for tablet d3dcac7c2d37456fb52797e62e01bdd9 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260503 13:58:05.040438 9511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3dcac7c2d37456fb52797e62e01bdd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.051510 9589 tablet_service.cc:1511] Processing CreateTablet for tablet b3b7ab9cca2847d392329f5c80956bbd (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260503 13:58:05.052776 9589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b7ab9cca2847d392329f5c80956bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.056660 9588 tablet_service.cc:1511] Processing CreateTablet for tablet cefaab375804454f943c82f15b8fd68b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260503 13:58:05.057864 9588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cefaab375804454f943c82f15b8fd68b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.062140 9587 tablet_service.cc:1511] Processing CreateTablet for tablet 83ebc0b241994520929db0a8b7fb8546 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260503 13:58:05.063391 9587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83ebc0b241994520929db0a8b7fb8546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.065887 9509 tablet_service.cc:1511] Processing CreateTablet for tablet c4a7792770d1446eafce219f5f28d7db (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260503 13:58:05.066120 9586 tablet_service.cc:1511] Processing CreateTablet for tablet 57b1804f75444731b8b3b6af3c038cf6 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260503 13:58:05.067153 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4a7792770d1446eafce219f5f28d7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.067333 9586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57b1804f75444731b8b3b6af3c038cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.070115 9585 tablet_service.cc:1511] Processing CreateTablet for tablet d3dcac7c2d37456fb52797e62e01bdd9 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260503 13:58:05.071336 9585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3dcac7c2d37456fb52797e62e01bdd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.074100 9584 tablet_service.cc:1511] Processing CreateTablet for tablet ddaccadb96de46d3a03ca8d2f3a8c6e0 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260503 13:58:05.075289 9584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddaccadb96de46d3a03ca8d2f3a8c6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.078042 9583 tablet_service.cc:1511] Processing CreateTablet for tablet c4a7792770d1446eafce219f5f28d7db (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260503 13:58:05.079263 9583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4a7792770d1446eafce219f5f28d7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.086504 9514 tablet_service.cc:1511] Processing CreateTablet for tablet 3046bdc111c5477eb4eabf09e7f4e755 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260503 13:58:05.087671 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3046bdc111c5477eb4eabf09e7f4e755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.089437 9515 tablet_service.cc:1511] Processing CreateTablet for tablet f5b937be606a421bab59946d15a81744 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260503 13:58:05.091216 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b937be606a421bab59946d15a81744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.103633 9513 tablet_service.cc:1511] Processing CreateTablet for tablet 22875ad0517447a2a9afc25bf367486f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260503 13:58:05.114776 9517 tablet_service.cc:1511] Processing CreateTablet for tablet b1dfd5295058420a9da1c1080e5caef3 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260503 13:58:05.116082 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1dfd5295058420a9da1c1080e5caef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.088282 9509 tablet_service.cc:1511] Processing CreateTablet for tablet 4d8e0a262a0c48d5b561d53f59fb808a (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260503 13:58:05.117426 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8e0a262a0c48d5b561d53f59fb808a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.126251 9510 tablet_service.cc:1511] Processing CreateTablet for tablet 7df5306f46674fdca645be800280031b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260503 13:58:05.129143 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22875ad0517447a2a9afc25bf367486f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.130337 9518 tablet_service.cc:1511] Processing CreateTablet for tablet facca7cb510c4c7a8be9484f3622012f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260503 13:58:05.137306 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet facca7cb510c4c7a8be9484f3622012f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.140862 9510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df5306f46674fdca645be800280031b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.158149 9442 tablet_service.cc:1511] Processing CreateTablet for tablet 497cd68663ac4d34b95230215aa13bd2 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260503 13:58:05.159426 9442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 497cd68663ac4d34b95230215aa13bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.161655 9443 tablet_service.cc:1511] Processing CreateTablet for tablet 32fd983d73354e649ff91e64f251e44f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260503 13:58:05.162837 9443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32fd983d73354e649ff91e64f251e44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.164011 9434 tablet_service.cc:1511] Processing CreateTablet for tablet c4a7792770d1446eafce219f5f28d7db (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260503 13:58:05.165376 9434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4a7792770d1446eafce219f5f28d7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.165680 9439 tablet_service.cc:1511] Processing CreateTablet for tablet cefaab375804454f943c82f15b8fd68b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260503 13:58:05.166917 9439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cefaab375804454f943c82f15b8fd68b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.166746 9438 tablet_service.cc:1511] Processing CreateTablet for tablet 83ebc0b241994520929db0a8b7fb8546 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260503 13:58:05.167922 9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83ebc0b241994520929db0a8b7fb8546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.169636 9437 tablet_service.cc:1511] Processing CreateTablet for tablet 57b1804f75444731b8b3b6af3c038cf6 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260503 13:58:05.170827 9437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57b1804f75444731b8b3b6af3c038cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.170650 9515 tablet_service.cc:1511] Processing CreateTablet for tablet 3b48fff133f84ff7827806460a23803b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260503 13:58:05.171787 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b48fff133f84ff7827806460a23803b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.162762 9435 tablet_service.cc:1511] Processing CreateTablet for tablet ddaccadb96de46d3a03ca8d2f3a8c6e0 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260503 13:58:05.173642 9435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddaccadb96de46d3a03ca8d2f3a8c6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.161674 9441 tablet_service.cc:1511] Processing CreateTablet for tablet ec9b187575714b4884a75ff6aa911e15 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260503 13:58:05.175438 9441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec9b187575714b4884a75ff6aa911e15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.165681 9440 tablet_service.cc:1511] Processing CreateTablet for tablet b3b7ab9cca2847d392329f5c80956bbd (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260503 13:58:05.178318 9440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b7ab9cca2847d392329f5c80956bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.173554 9436 tablet_service.cc:1511] Processing CreateTablet for tablet d3dcac7c2d37456fb52797e62e01bdd9 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260503 13:58:05.181658 9436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3dcac7c2d37456fb52797e62e01bdd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.202791 9589 tablet_service.cc:1511] Processing CreateTablet for tablet f5b937be606a421bab59946d15a81744 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260503 13:58:05.204059 9589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b937be606a421bab59946d15a81744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.215248 9585 tablet_service.cc:1511] Processing CreateTablet for tablet 3046bdc111c5477eb4eabf09e7f4e755 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260503 13:58:05.216471 9585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3046bdc111c5477eb4eabf09e7f4e755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.221587 9583 tablet_service.cc:1511] Processing CreateTablet for tablet 4d8e0a262a0c48d5b561d53f59fb808a (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260503 13:58:05.221763 9590 tablet_service.cc:1511] Processing CreateTablet for tablet 22875ad0517447a2a9afc25bf367486f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260503 13:58:05.230199 9440 tablet_service.cc:1511] Processing CreateTablet for tablet 306069d785d24834b84d590925e644cf (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260503 13:58:05.193594 9514 tablet_service.cc:1511] Processing CreateTablet for tablet 38f7f4127357496eada52ad754f934e1 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260503 13:58:05.242421 9642 tablet_bootstrap.cc:492] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:05.238325 9516 tablet_service.cc:1511] Processing CreateTablet for tablet b07d47a095894b958145d029c84c9123 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260503 13:58:05.245193 9644 tablet_bootstrap.cc:492] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:05.245605 9510 tablet_service.cc:1511] Processing CreateTablet for tablet 54bf4ee675fb4fcf800b317817906dcc (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260503 13:58:05.249135 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f7f4127357496eada52ad754f934e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.274097 9440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306069d785d24834b84d590925e644cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.284070 9642 tablet_bootstrap.cc:654] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:05.289783 9591 tablet_service.cc:1511] Processing CreateTablet for tablet b1dfd5295058420a9da1c1080e5caef3 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260503 13:58:05.293599 9584 tablet_service.cc:1511] Processing CreateTablet for tablet 7df5306f46674fdca645be800280031b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260503 13:58:05.294322 9589 tablet_service.cc:1511] Processing CreateTablet for tablet 38f7f4127357496eada52ad754f934e1 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260503 13:58:05.287590 9590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22875ad0517447a2a9afc25bf367486f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.289814 9438 tablet_service.cc:1511] Processing CreateTablet for tablet b1dfd5295058420a9da1c1080e5caef3 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260503 13:58:05.299093 9643 tablet_bootstrap.cc:492] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:05.299504 9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1dfd5295058420a9da1c1080e5caef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.304219 9584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df5306f46674fdca645be800280031b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.308797 9591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1dfd5295058420a9da1c1080e5caef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.309167 9434 tablet_service.cc:1511] Processing CreateTablet for tablet 4d8e0a262a0c48d5b561d53f59fb808a (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260503 13:58:05.309365 9585 tablet_service.cc:1511] Processing CreateTablet for tablet 3b48fff133f84ff7827806460a23803b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260503 13:58:05.310427 9434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8e0a262a0c48d5b561d53f59fb808a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.313052 9588 tablet_service.cc:1511] Processing CreateTablet for tablet b07d47a095894b958145d029c84c9123 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260503 13:58:05.317076 9511 tablet_service.cc:1511] Processing CreateTablet for tablet a17efa5a2f524bcb966ff62c4182503f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260503 13:58:05.318387 9511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17efa5a2f524bcb966ff62c4182503f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.321681 9510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54bf4ee675fb4fcf800b317817906dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.318392 9586 tablet_service.cc:1511] Processing CreateTablet for tablet facca7cb510c4c7a8be9484f3622012f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260503 13:58:05.325827 9515 tablet_service.cc:1511] Processing CreateTablet for tablet 05ba056a4d0c406a881cc34cb66078bc (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260503 13:58:05.326444 9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b07d47a095894b958145d029c84c9123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.327466 9512 tablet_service.cc:1511] Processing CreateTablet for tablet 2c918e4ac84c4238a06220bc2817286f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260503 13:58:05.329819 9517 tablet_service.cc:1511] Processing CreateTablet for tablet 3500374093e84b968ed96ed4dbdfda69 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260503 13:58:05.331684 9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c918e4ac84c4238a06220bc2817286f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.325917 9513 tablet_service.cc:1511] Processing CreateTablet for tablet a763e2ddf6e7463bb123ce8d870a1afb (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260503 13:58:05.316351 9437 tablet_service.cc:1511] Processing CreateTablet for tablet 22875ad0517447a2a9afc25bf367486f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260503 13:58:05.334007 9437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22875ad0517447a2a9afc25bf367486f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.335335 9518 tablet_service.cc:1511] Processing CreateTablet for tablet 0db0ab3576734ec18108a10f65e31341 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260503 13:58:05.336012 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3500374093e84b968ed96ed4dbdfda69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.337019 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a763e2ddf6e7463bb123ce8d870a1afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.347958 9644 tablet_bootstrap.cc:654] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:05.348711 9643 tablet_bootstrap.cc:654] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:05.350581 9439 tablet_service.cc:1511] Processing CreateTablet for tablet 3046bdc111c5477eb4eabf09e7f4e755 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260503 13:58:05.351758 9439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3046bdc111c5477eb4eabf09e7f4e755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.355926 9435 tablet_service.cc:1511] Processing CreateTablet for tablet 7df5306f46674fdca645be800280031b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260503 13:58:05.363452 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ba056a4d0c406a881cc34cb66078bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.364720 9585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b48fff133f84ff7827806460a23803b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.371493 9436 tablet_service.cc:1511] Processing CreateTablet for tablet f5b937be606a421bab59946d15a81744 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260503 13:58:05.371969 9442 tablet_service.cc:1511] Processing CreateTablet for tablet facca7cb510c4c7a8be9484f3622012f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260503 13:58:05.376842 9583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8e0a262a0c48d5b561d53f59fb808a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.377262 9514 tablet_service.cc:1511] Processing CreateTablet for tablet 306069d785d24834b84d590925e644cf (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260503 13:58:05.378436 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306069d785d24834b84d590925e644cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.370580 9592 tablet_service.cc:1511] Processing CreateTablet for tablet a17efa5a2f524bcb966ff62c4182503f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260503 13:58:05.384378 9441 tablet_service.cc:1511] Processing CreateTablet for tablet 38f7f4127357496eada52ad754f934e1 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260503 13:58:05.385253 9592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17efa5a2f524bcb966ff62c4182503f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.385413 9509 tablet_service.cc:1511] Processing CreateTablet for tablet 50d2bf9820c443d5b7d017429cb0a818 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260503 13:58:05.386672 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50d2bf9820c443d5b7d017429cb0a818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.388805 9589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f7f4127357496eada52ad754f934e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.391573 9435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df5306f46674fdca645be800280031b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.391793 9443 tablet_service.cc:1511] Processing CreateTablet for tablet 3b48fff133f84ff7827806460a23803b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260503 13:58:05.392827 9436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b937be606a421bab59946d15a81744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.391809 9587 tablet_service.cc:1511] Processing CreateTablet for tablet 54bf4ee675fb4fcf800b317817906dcc (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260503 13:58:05.394105 9443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b48fff133f84ff7827806460a23803b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.400730 9586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet facca7cb510c4c7a8be9484f3622012f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.403800 9591 tablet_service.cc:1511] Processing CreateTablet for tablet 05ba056a4d0c406a881cc34cb66078bc (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260503 13:58:05.404503 9442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet facca7cb510c4c7a8be9484f3622012f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.404995 9591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ba056a4d0c406a881cc34cb66078bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.407209 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db0ab3576734ec18108a10f65e31341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.420851 9587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54bf4ee675fb4fcf800b317817906dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.423943 9514 tablet_service.cc:1511] Processing CreateTablet for tablet d5c2e56fa22a496280f2678cd6657c3d (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260503 13:58:05.424876 9588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b07d47a095894b958145d029c84c9123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.425171 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5c2e56fa22a496280f2678cd6657c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.433933 9643 tablet_bootstrap.cc:492] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:05.434393 9643 ts_tablet_manager.cc:1403] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.140s user 0.011s sys 0.000s
I20260503 13:58:05.437124 9643 raft_consensus.cc:359] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:05.437780 9643 raft_consensus.cc:385] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:05.438058 9643 raft_consensus.cc:740] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:05.438839 9643 consensus_queue.cc:260] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:05.439364 9441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f7f4127357496eada52ad754f934e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.449638 9440 tablet_service.cc:1511] Processing CreateTablet for tablet b07d47a095894b958145d029c84c9123 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260503 13:58:05.451087 9440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b07d47a095894b958145d029c84c9123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.449719 9591 tablet_service.cc:1511] Processing CreateTablet for tablet 2804950dfaaa4c1aa2da4b8c47dbd97f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260503 13:58:05.453734 9591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2804950dfaaa4c1aa2da4b8c47dbd97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.461611 9516 tablet_service.cc:1511] Processing CreateTablet for tablet 8b9644d56a7f4556ae897658749f02e4 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260503 13:58:05.462889 9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9644d56a7f4556ae897658749f02e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.466050 9439 tablet_service.cc:1511] Processing CreateTablet for tablet 54bf4ee675fb4fcf800b317817906dcc (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260503 13:58:05.467207 9439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54bf4ee675fb4fcf800b317817906dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.477758 9642 tablet_bootstrap.cc:492] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:05.478307 9642 ts_tablet_manager.cc:1403] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.282s user 0.025s sys 0.022s
I20260503 13:58:05.487803 9585 tablet_service.cc:1511] Processing CreateTablet for tablet d5c2e56fa22a496280f2678cd6657c3d (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260503 13:58:05.489065 9585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5c2e56fa22a496280f2678cd6657c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.496757 9627 heartbeater.cc:499] Master 127.8.26.254:40413 was elected leader, sending a full tablet report...
I20260503 13:58:05.497581 9512 tablet_service.cc:1511] Processing CreateTablet for tablet 63e583b228f24d97b0e2b31046675c77 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260503 13:58:05.498854 9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63e583b228f24d97b0e2b31046675c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.501904 9644 tablet_bootstrap.cc:492] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:05.502341 9644 ts_tablet_manager.cc:1403] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.257s user 0.011s sys 0.037s
I20260503 13:58:05.503504 9588 tablet_service.cc:1511] Processing CreateTablet for tablet 8b9644d56a7f4556ae897658749f02e4 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260503 13:58:05.504721 9588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9644d56a7f4556ae897658749f02e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.505862 9513 tablet_service.cc:1511] Processing CreateTablet for tablet 23732c2c1b724f65b22c4332efbd7da2 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260503 13:58:05.507086 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23732c2c1b724f65b22c4332efbd7da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.503504 9591 tablet_service.cc:1511] Processing CreateTablet for tablet 2c918e4ac84c4238a06220bc2817286f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260503 13:58:05.509016 9591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c918e4ac84c4238a06220bc2817286f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.510463 9644 raft_consensus.cc:359] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:05.511173 9644 raft_consensus.cc:385] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:05.511430 9644 raft_consensus.cc:740] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:05.512032 9644 consensus_queue.cc:260] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:05.513376 9512 tablet_service.cc:1511] Processing CreateTablet for tablet 973019c2128c4039aa1d442714f23ae7 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260503 13:58:05.514531 9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 973019c2128c4039aa1d442714f23ae7. 1 dirs total, 0 dirs full, 0 dirs failed
W20260503 13:58:05.560745 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:05.630947 9511 tablet_service.cc:1511] Processing CreateTablet for tablet 8cc630de8e5547ceb5c178981ee99fc2 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260503 13:58:05.632262 9511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cc630de8e5547ceb5c178981ee99fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.633558 9514 tablet_service.cc:1511] Processing CreateTablet for tablet 2804950dfaaa4c1aa2da4b8c47dbd97f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260503 13:58:05.578822 9434 tablet_service.cc:1511] Processing CreateTablet for tablet a17efa5a2f524bcb966ff62c4182503f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260503 13:58:05.590775 9437 tablet_service.cc:1511] Processing CreateTablet for tablet 05ba056a4d0c406a881cc34cb66078bc (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260503 13:58:05.636318 9515 tablet_service.cc:1511] Processing CreateTablet for tablet 09c4dc2df168460e83f4f02b17667151 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260503 13:58:05.637511 9437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ba056a4d0c406a881cc34cb66078bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.637642 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c4dc2df168460e83f4f02b17667151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.594890 9438 tablet_service.cc:1511] Processing CreateTablet for tablet a763e2ddf6e7463bb123ce8d870a1afb (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260503 13:58:05.595608 9436 tablet_service.cc:1511] Processing CreateTablet for tablet 2c918e4ac84c4238a06220bc2817286f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260503 13:58:05.640905 9436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c918e4ac84c4238a06220bc2817286f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.641026 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2804950dfaaa4c1aa2da4b8c47dbd97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.645164 9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a763e2ddf6e7463bb123ce8d870a1afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.600456 9441 tablet_service.cc:1511] Processing CreateTablet for tablet 3500374093e84b968ed96ed4dbdfda69 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260503 13:58:05.607316 9435 tablet_service.cc:1511] Processing CreateTablet for tablet 0db0ab3576734ec18108a10f65e31341 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260503 13:58:05.650046 9435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db0ab3576734ec18108a10f65e31341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.651968 9441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3500374093e84b968ed96ed4dbdfda69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.657612 9511 tablet_service.cc:1511] Processing CreateTablet for tablet 6d03f17c8e3a48fcbbb01ac5504d81fe (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260503 13:58:05.659173 9511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d03f17c8e3a48fcbbb01ac5504d81fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.671865 9517 tablet_service.cc:1511] Processing CreateTablet for tablet 26449f3ab09141ada50d9334532cb3ec (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260503 13:58:05.673250 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26449f3ab09141ada50d9334532cb3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.673705 9511 tablet_service.cc:1511] Processing CreateTablet for tablet 322e401806f7452c9714ce855e35ca9b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260503 13:58:05.674830 9511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 322e401806f7452c9714ce855e35ca9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.519445 9583 tablet_service.cc:1511] Processing CreateTablet for tablet 0db0ab3576734ec18108a10f65e31341 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260503 13:58:05.686637 9583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db0ab3576734ec18108a10f65e31341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.689522 9584 tablet_service.cc:1511] Processing CreateTablet for tablet a763e2ddf6e7463bb123ce8d870a1afb (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260503 13:58:05.690877 9584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a763e2ddf6e7463bb123ce8d870a1afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.691771 9434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17efa5a2f524bcb966ff62c4182503f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.698031 9553 heartbeater.cc:499] Master 127.8.26.254:40413 was elected leader, sending a full tablet report...
I20260503 13:58:05.528426 9590 tablet_service.cc:1511] Processing CreateTablet for tablet 50d2bf9820c443d5b7d017429cb0a818 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260503 13:58:05.527185 9592 tablet_service.cc:1511] Processing CreateTablet for tablet 3500374093e84b968ed96ed4dbdfda69 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
W20260503 13:58:05.611251 9479 tablet.cc:2404] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260503 13:58:05.614885 9443 tablet_service.cc:1511] Processing CreateTablet for tablet d5c2e56fa22a496280f2678cd6657c3d (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260503 13:58:05.619896 9439 tablet_service.cc:1511] Processing CreateTablet for tablet 8b9644d56a7f4556ae897658749f02e4 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260503 13:58:05.622342 9440 tablet_service.cc:1511] Processing CreateTablet for tablet 2804950dfaaa4c1aa2da4b8c47dbd97f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260503 13:58:05.624594 9510 tablet_service.cc:1511] Processing CreateTablet for tablet 010e981b4bda450e9e11c3f6ff93f7eb (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260503 13:58:05.529444 9643 ts_tablet_manager.cc:1434] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.095s user 0.026s sys 0.025s
I20260503 13:58:05.610282 9478 heartbeater.cc:499] Master 127.8.26.254:40413 was elected leader, sending a full tablet report...
I20260503 13:58:05.608522 9442 tablet_service.cc:1511] Processing CreateTablet for tablet 50d2bf9820c443d5b7d017429cb0a818 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260503 13:58:05.526851 9591 tablet_service.cc:1511] Processing CreateTablet for tablet 306069d785d24834b84d590925e644cf (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260503 13:58:05.708098 9585 tablet_service.cc:1511] Processing CreateTablet for tablet 09c4dc2df168460e83f4f02b17667151 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260503 13:58:05.526000 9642 raft_consensus.cc:359] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:05.720947 9642 raft_consensus.cc:385] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:05.721261 9642 raft_consensus.cc:740] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:05.721999 9642 consensus_queue.cc:260] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:05.520983 9644 ts_tablet_manager.cc:1434] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.018s user 0.009s sys 0.002s
I20260503 13:58:05.702541 9590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50d2bf9820c443d5b7d017429cb0a818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.724345 9592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3500374093e84b968ed96ed4dbdfda69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.727716 9510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010e981b4bda450e9e11c3f6ff93f7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.734054 9643 tablet_bootstrap.cc:492] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:05.708155 9518 tablet_service.cc:1511] Processing CreateTablet for tablet 1c480795fed14481acd9ca2d189b9583 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260503 13:58:05.709517 9649 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:05.735291 9649 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:05.737277 9591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306069d785d24834b84d590925e644cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.737874 9516 tablet_service.cc:1511] Processing CreateTablet for tablet a74decf860264836999f054535dd35ab (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260503 13:58:05.717883 9443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5c2e56fa22a496280f2678cd6657c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.740815 9439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9644d56a7f4556ae897658749f02e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.743492 9440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2804950dfaaa4c1aa2da4b8c47dbd97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.745420 9649 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:05.751179 9435 tablet_service.cc:1511] Processing CreateTablet for tablet 010e981b4bda450e9e11c3f6ff93f7eb (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260503 13:58:05.753989 9588 tablet_service.cc:1511] Processing CreateTablet for tablet 23732c2c1b724f65b22c4332efbd7da2 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260503 13:58:05.755147 9588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23732c2c1b724f65b22c4332efbd7da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.758091 9587 tablet_service.cc:1511] Processing CreateTablet for tablet 63e583b228f24d97b0e2b31046675c77 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260503 13:58:05.759325 9587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63e583b228f24d97b0e2b31046675c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.759030 9589 tablet_service.cc:1511] Processing CreateTablet for tablet 973019c2128c4039aa1d442714f23ae7 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260503 13:58:05.760267 9589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 973019c2128c4039aa1d442714f23ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.752329 9435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010e981b4bda450e9e11c3f6ff93f7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.760617 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c480795fed14481acd9ca2d189b9583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.767907 9644 tablet_bootstrap.cc:492] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:05.771735 9583 tablet_service.cc:1511] Processing CreateTablet for tablet 8cc630de8e5547ceb5c178981ee99fc2 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260503 13:58:05.728129 9514 tablet_service.cc:1511] Processing CreateTablet for tablet 9b9eaf6f8db04058b132b139d110dd24 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260503 13:58:05.776794 9586 tablet_service.cc:1511] Processing CreateTablet for tablet 010e981b4bda450e9e11c3f6ff93f7eb (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260503 13:58:05.774348 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9eaf6f8db04058b132b139d110dd24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.792959 9583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cc630de8e5547ceb5c178981ee99fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.793764 9509 tablet_service.cc:1511] Processing CreateTablet for tablet d049f50b1b5745d3b2e3568ed86c16d8 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260503 13:58:05.794275 9442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50d2bf9820c443d5b7d017429cb0a818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.794950 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d049f50b1b5745d3b2e3568ed86c16d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.798605 9437 tablet_service.cc:1511] Processing CreateTablet for tablet 09c4dc2df168460e83f4f02b17667151 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260503 13:58:05.805903 9585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c4dc2df168460e83f4f02b17667151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.807178 9643 tablet_bootstrap.cc:654] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:05.810779 9517 tablet_service.cc:1511] Processing CreateTablet for tablet c6b3066cc4e4411f91d1cde17a952170 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260503 13:58:05.813768 9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a74decf860264836999f054535dd35ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.816354 9586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010e981b4bda450e9e11c3f6ff93f7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.845108 9437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c4dc2df168460e83f4f02b17667151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.850122 9440 tablet_service.cc:1511] Processing CreateTablet for tablet 973019c2128c4039aa1d442714f23ae7 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260503 13:58:05.850023 9436 tablet_service.cc:1511] Processing CreateTablet for tablet 23732c2c1b724f65b22c4332efbd7da2 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260503 13:58:05.853075 9439 tablet_service.cc:1511] Processing CreateTablet for tablet 63e583b228f24d97b0e2b31046675c77 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260503 13:58:05.853976 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6b3066cc4e4411f91d1cde17a952170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.855216 9584 tablet_service.cc:1511] Processing CreateTablet for tablet 6d03f17c8e3a48fcbbb01ac5504d81fe (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260503 13:58:05.856357 9584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d03f17c8e3a48fcbbb01ac5504d81fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.859234 9439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63e583b228f24d97b0e2b31046675c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.860155 9441 tablet_service.cc:1511] Processing CreateTablet for tablet 8cc630de8e5547ceb5c178981ee99fc2 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260503 13:58:05.861179 9436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23732c2c1b724f65b22c4332efbd7da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.819896 9511 tablet_service.cc:1511] Processing CreateTablet for tablet 0f951d26e4ed4c3bb307eb276954663b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260503 13:58:05.869930 9589 tablet_service.cc:1511] Processing CreateTablet for tablet 322e401806f7452c9714ce855e35ca9b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260503 13:58:05.875362 9589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 322e401806f7452c9714ce855e35ca9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.876210 9642 ts_tablet_manager.cc:1434] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.398s user 0.011s sys 0.010s
I20260503 13:58:05.821942 9513 tablet_service.cc:1511] Processing CreateTablet for tablet f3d3814bacf748a4928419aa5e860f4a (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260503 13:58:05.818146 9515 tablet_service.cc:1511] Processing CreateTablet for tablet 11b8ecb173e84681b8fb389c89395838 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260503 13:58:05.818138 9512 tablet_service.cc:1511] Processing CreateTablet for tablet b633f30c57174316b12d468e847ca831 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260503 13:58:05.884570 9440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 973019c2128c4039aa1d442714f23ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.896793 9642 tablet_bootstrap.cc:492] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:05.896615 9510 tablet_service.cc:1511] Processing CreateTablet for tablet 96fdbf26d7134a238a95b243fe5b8aac (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260503 13:58:05.905051 9441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cc630de8e5547ceb5c178981ee99fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.908809 9511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f951d26e4ed4c3bb307eb276954663b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.909379 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11b8ecb173e84681b8fb389c89395838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.912539 9510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fdbf26d7134a238a95b243fe5b8aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.913302 9588 tablet_service.cc:1511] Processing CreateTablet for tablet 1c480795fed14481acd9ca2d189b9583 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260503 13:58:05.914502 9588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c480795fed14481acd9ca2d189b9583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.916432 9642 tablet_bootstrap.cc:654] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:05.917352 9644 tablet_bootstrap.cc:654] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:05.917862 9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b633f30c57174316b12d468e847ca831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.928790 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3d3814bacf748a4928419aa5e860f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.811280 9592 tablet_service.cc:1511] Processing CreateTablet for tablet 26449f3ab09141ada50d9334532cb3ec (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260503 13:58:05.939656 9514 tablet_service.cc:1511] Processing CreateTablet for tablet 94b14392ab2e47fea655f24d4c3e5da5 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260503 13:58:05.942170 9591 tablet_service.cc:1511] Processing CreateTablet for tablet 9b9eaf6f8db04058b132b139d110dd24 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260503 13:58:05.943364 9591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9eaf6f8db04058b132b139d110dd24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.943418 9515 tablet_service.cc:1511] Processing CreateTablet for tablet 8650e2ab26ac47f8a5a6cd4dd6c30d50 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260503 13:58:05.944757 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8650e2ab26ac47f8a5a6cd4dd6c30d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.963500 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:05.965373 9566 leader_election.cc:343] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:05.966355 9587 tablet_service.cc:1511] Processing CreateTablet for tablet d049f50b1b5745d3b2e3568ed86c16d8 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260503 13:58:05.967607 9587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d049f50b1b5745d3b2e3568ed86c16d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.969568 9518 tablet_service.cc:1511] Processing CreateTablet for tablet 5b4c6d7380b242c2930533560310a3f4 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260503 13:58:05.970767 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b4c6d7380b242c2930533560310a3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.971583 9592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26449f3ab09141ada50d9334532cb3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.975142 9584 tablet_service.cc:1511] Processing CreateTablet for tablet c6b3066cc4e4411f91d1cde17a952170 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260503 13:58:05.957163 9585 tablet_service.cc:1511] Processing CreateTablet for tablet a74decf860264836999f054535dd35ab (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260503 13:58:05.972901 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b14392ab2e47fea655f24d4c3e5da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.980546 9434 tablet_service.cc:1511] Processing CreateTablet for tablet 26449f3ab09141ada50d9334532cb3ec (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260503 13:58:05.987417 9434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26449f3ab09141ada50d9334532cb3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.989995 9584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6b3066cc4e4411f91d1cde17a952170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.991791 9585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a74decf860264836999f054535dd35ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.994848 9443 tablet_service.cc:1511] Processing CreateTablet for tablet 6d03f17c8e3a48fcbbb01ac5504d81fe (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260503 13:58:05.996157 9443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d03f17c8e3a48fcbbb01ac5504d81fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.001327 9438 tablet_service.cc:1511] Processing CreateTablet for tablet 322e401806f7452c9714ce855e35ca9b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260503 13:58:06.002485 9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 322e401806f7452c9714ce855e35ca9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.003841 9650 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.004279 9650 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.006673 9650 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:06.006402 9442 tablet_service.cc:1511] Processing CreateTablet for tablet 1c480795fed14481acd9ca2d189b9583 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260503 13:58:06.024816 9651 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.025312 9651 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.026041 9517 tablet_service.cc:1511] Processing CreateTablet for tablet d04573f959de406a9590a1603f633e7e (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260503 13:58:06.026742 9589 tablet_service.cc:1511] Processing CreateTablet for tablet 11b8ecb173e84681b8fb389c89395838 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260503 13:58:06.027262 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d04573f959de406a9590a1603f633e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.027897 9589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11b8ecb173e84681b8fb389c89395838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:05.993146 9586 tablet_service.cc:1511] Processing CreateTablet for tablet b633f30c57174316b12d468e847ca831 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260503 13:58:06.029105 9651 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:06.041034 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:06.041733 9528 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:06.043200 9567 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:06.050123 9649 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:06.050470 9649 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:06.051659 9649 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.053361 9512 tablet_service.cc:1511] Processing CreateTablet for tablet a5439e0e393b4112a26459eb56502fae (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260503 13:58:06.054507 9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5439e0e393b4112a26459eb56502fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.056986 9440 tablet_service.cc:1511] Processing CreateTablet for tablet 9b9eaf6f8db04058b132b139d110dd24 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260503 13:58:06.059048 9590 tablet_service.cc:1511] Processing CreateTablet for tablet 0f951d26e4ed4c3bb307eb276954663b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260503 13:58:06.066339 9437 tablet_service.cc:1511] Processing CreateTablet for tablet a74decf860264836999f054535dd35ab (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260503 13:58:06.068148 9513 tablet_service.cc:1511] Processing CreateTablet for tablet 030fcc1b19c3489ea0ca320576ed54ee (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260503 13:58:06.069413 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030fcc1b19c3489ea0ca320576ed54ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.045138 9442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c480795fed14481acd9ca2d189b9583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.064352 9509 tablet_service.cc:1511] Processing CreateTablet for tablet 5c55d864e0b0416ebdc09ce6e0310a8f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260503 13:58:06.095378 9586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b633f30c57174316b12d468e847ca831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.103013 9649 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.104120 9516 tablet_service.cc:1511] Processing CreateTablet for tablet 754ac001a92e4dd6a0b6620803b83722 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260503 13:58:06.105655 9649 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:06.106035 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:06.106699 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:06.115137 9437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a74decf860264836999f054535dd35ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.115741 9440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9eaf6f8db04058b132b139d110dd24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.112816 9528 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:58:06.119728 9566 leader_election.cc:343] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.125504 9590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f951d26e4ed4c3bb307eb276954663b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.126499 9441 tablet_service.cc:1511] Processing CreateTablet for tablet d049f50b1b5745d3b2e3568ed86c16d8 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260503 13:58:06.127700 9441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d049f50b1b5745d3b2e3568ed86c16d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.129314 9583 tablet_service.cc:1511] Processing CreateTablet for tablet 94b14392ab2e47fea655f24d4c3e5da5 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260503 13:58:06.129649 9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 754ac001a92e4dd6a0b6620803b83722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.130456 9583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b14392ab2e47fea655f24d4c3e5da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.139963 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c55d864e0b0416ebdc09ce6e0310a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.143527 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:06.145103 9418 leader_election.cc:343] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.145632 9642 tablet_bootstrap.cc:492] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:06.146122 9642 ts_tablet_manager.cc:1403] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.250s user 0.022s sys 0.023s
I20260503 13:58:06.150085 9643 tablet_bootstrap.cc:492] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:06.150544 9643 ts_tablet_manager.cc:1403] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.417s user 0.024s sys 0.024s
I20260503 13:58:06.153121 9436 tablet_service.cc:1511] Processing CreateTablet for tablet b633f30c57174316b12d468e847ca831 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260503 13:58:06.153355 9643 raft_consensus.cc:359] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.154026 9643 raft_consensus.cc:385] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.154309 9436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b633f30c57174316b12d468e847ca831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.154316 9643 raft_consensus.cc:740] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.155205 9643 consensus_queue.cc:260] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.156414 9642 raft_consensus.cc:359] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.157104 9642 raft_consensus.cc:385] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.157387 9642 raft_consensus.cc:740] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.158213 9642 consensus_queue.cc:260] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.117118 9435 tablet_service.cc:1511] Processing CreateTablet for tablet c6b3066cc4e4411f91d1cde17a952170 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260503 13:58:06.159646 9643 ts_tablet_manager.cc:1434] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260503 13:58:06.159811 9435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6b3066cc4e4411f91d1cde17a952170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.160622 9643 tablet_bootstrap.cc:492] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:06.162606 9642 ts_tablet_manager.cc:1434] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20260503 13:58:06.163481 9642 tablet_bootstrap.cc:492] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:06.178661 9588 tablet_service.cc:1511] Processing CreateTablet for tablet f3d3814bacf748a4928419aa5e860f4a (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260503 13:58:06.179848 9588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3d3814bacf748a4928419aa5e860f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.155782 9528 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:06.185498 9567 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:06.186507 9649 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:06.193041 9644 tablet_bootstrap.cc:492] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:06.193514 9644 ts_tablet_manager.cc:1403] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.468s user 0.019s sys 0.018s
I20260503 13:58:06.194614 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:06.196125 9416 leader_election.cc:343] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.196537 9416 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:06.197698 9650 raft_consensus.cc:2749] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:06.183622 9439 tablet_service.cc:1511] Processing CreateTablet for tablet 11b8ecb173e84681b8fb389c89395838 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260503 13:58:06.214089 9439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11b8ecb173e84681b8fb389c89395838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.210109 9643 tablet_bootstrap.cc:654] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.222023 9443 tablet_service.cc:1511] Processing CreateTablet for tablet 0f951d26e4ed4c3bb307eb276954663b (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260503 13:58:06.196472 9644 raft_consensus.cc:359] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.222926 9441 tablet_service.cc:1511] Processing CreateTablet for tablet 96fdbf26d7134a238a95b243fe5b8aac (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260503 13:58:06.225699 9584 tablet_service.cc:1511] Processing CreateTablet for tablet a5439e0e393b4112a26459eb56502fae (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260503 13:58:06.239786 9584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5439e0e393b4112a26459eb56502fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.223994 9586 tablet_service.cc:1511] Processing CreateTablet for tablet 754ac001a92e4dd6a0b6620803b83722 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260503 13:58:06.240392 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:06.241195 9644 raft_consensus.cc:385] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.241516 9644 raft_consensus.cc:740] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.215673 9434 tablet_service.cc:1511] Processing CreateTablet for tablet f3d3814bacf748a4928419aa5e860f4a (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
W20260503 13:58:06.241815 9492 leader_election.cc:343] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.242312 9644 consensus_queue.cc:260] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.229413 9585 tablet_service.cc:1511] Processing CreateTablet for tablet 96fdbf26d7134a238a95b243fe5b8aac (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260503 13:58:06.223320 9443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f951d26e4ed4c3bb307eb276954663b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.247255 9586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 754ac001a92e4dd6a0b6620803b83722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.248256 9434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3d3814bacf748a4928419aa5e860f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.248637 9585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fdbf26d7134a238a95b243fe5b8aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.220301 9642 tablet_bootstrap.cc:654] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.227581 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:06.252139 9441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fdbf26d7134a238a95b243fe5b8aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.261819 9442 tablet_service.cc:1511] Processing CreateTablet for tablet a5439e0e393b4112a26459eb56502fae (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260503 13:58:06.262526 9438 tablet_service.cc:1511] Processing CreateTablet for tablet 8650e2ab26ac47f8a5a6cd4dd6c30d50 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260503 13:58:06.262984 9442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5439e0e393b4112a26459eb56502fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.263635 9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8650e2ab26ac47f8a5a6cd4dd6c30d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.249496 9649 raft_consensus.cc:697] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:06.265290 9643 tablet_bootstrap.cc:492] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:06.265764 9643 ts_tablet_manager.cc:1403] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.105s user 0.007s sys 0.003s
I20260503 13:58:06.267966 9583 tablet_service.cc:1511] Processing CreateTablet for tablet 5b4c6d7380b242c2930533560310a3f4 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260503 13:58:06.268493 9643 raft_consensus.cc:359] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.269214 9643 raft_consensus.cc:385] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.269492 9643 raft_consensus.cc:740] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.270150 9643 consensus_queue.cc:260] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.274735 9643 ts_tablet_manager.cc:1434] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260503 13:58:06.275007 9584 tablet_service.cc:1511] Processing CreateTablet for tablet 5c55d864e0b0416ebdc09ce6e0310a8f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260503 13:58:06.275678 9643 tablet_bootstrap.cc:492] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:06.276361 9584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c55d864e0b0416ebdc09ce6e0310a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.279109 9438 tablet_service.cc:1511] Processing CreateTablet for tablet 5b4c6d7380b242c2930533560310a3f4 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260503 13:58:06.280385 9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b4c6d7380b242c2930533560310a3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.265230 9649 consensus_queue.cc:237] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.285735 9587 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43746 (request call id 44) took 1001 ms. Trace:
I20260503 13:58:06.286193 9587 rpcz_store.cc:276] 0503 13:58:05.284120 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:05.965415 (+681295us) service_pool.cc:224] Handling call
0503 13:58:05.967162 (+ 1747us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:05.967256 (+ 94us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.093173 (+125917us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.285210 (+192037us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.285680 (+ 470us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:06.292021 9440 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46706 (request call id 42) took 1014 ms. Trace:
I20260503 13:58:06.232336 9437 tablet_service.cc:1511] Processing CreateTablet for tablet 94b14392ab2e47fea655f24d4c3e5da5 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260503 13:58:06.286192 9435 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46706 (request call id 45) took 1007 ms. Trace:
I20260503 13:58:06.292953 9435 rpcz_store.cc:276] 0503 13:58:05.278845 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.112265 (+833420us) service_pool.cc:224] Handling call
0503 13:58:06.159393 (+ 47128us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.159481 (+ 88us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.203139 (+ 43658us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.285723 (+ 82584us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.286138 (+ 415us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:06.295151 9583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b4c6d7380b242c2930533560310a3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.296674 9643 tablet_bootstrap.cc:654] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.298429 9651 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.298804 9651 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.301401 9589 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43746 (request call id 47) took 1003 ms. Trace:
I20260503 13:58:06.302151 9651 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:06.301826 9589 rpcz_store.cc:276] 0503 13:58:05.298465 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.025889 (+727424us) service_pool.cc:224] Handling call
0503 13:58:06.027482 (+ 1593us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.027570 (+ 88us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.203168 (+175598us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.300942 (+ 97774us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.301349 (+ 407us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:06.302850 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:06.303411 9453 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:06.304591 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:06.304986 9437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b14392ab2e47fea655f24d4c3e5da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.305210 9492 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:06.305236 9601 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:06.288322 9587 tablet_service.cc:1511] Processing CreateTablet for tablet d04573f959de406a9590a1603f633e7e (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260503 13:58:06.306027 9651 raft_consensus.cc:2804] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:06.306392 9651 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:06.306712 9651 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.301319 9590 tablet_service.cc:1511] Processing CreateTablet for tablet 030fcc1b19c3489ea0ca320576ed54ee (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260503 13:58:06.312002 9442 tablet_service.cc:1511] Processing CreateTablet for tablet 030fcc1b19c3489ea0ca320576ed54ee (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260503 13:58:06.312704 9438 tablet_service.cc:1511] Processing CreateTablet for tablet d04573f959de406a9590a1603f633e7e (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260503 13:58:06.313203 9651 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.314409 9436 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46706 (request call id 46) took 1027 ms. Trace:
I20260503 13:58:06.320771 9436 rpcz_store.cc:276] 0503 13:58:05.292453 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.152197 (+859744us) service_pool.cc:224] Handling call
0503 13:58:06.153891 (+ 1694us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.153982 (+ 91us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.253700 (+ 99718us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.313955 (+ 60255us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.314356 (+ 401us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:06.315615 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:06.321945 9453 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.322084 9644 ts_tablet_manager.cc:1434] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.128s user 0.010s sys 0.005s
I20260503 13:58:06.305357 9435 tablet_service.cc:1511] Processing CreateTablet for tablet 5c55d864e0b0416ebdc09ce6e0310a8f (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260503 13:58:06.323135 9644 tablet_bootstrap.cc:492] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:06.315984 9665 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.325050 9665 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.326068 9443 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46706 (request call id 48) took 1021 ms. Trace:
I20260503 13:58:06.327127 9665 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:06.329738 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:06.330368 9452 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:06.332067 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:06.292389 9440 rpcz_store.cc:276] 0503 13:58:05.277898 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.055955 (+778057us) service_pool.cc:224] Handling call
0503 13:58:06.057736 (+ 1781us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.057816 (+ 80us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.168094 (+110278us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.291434 (+123340us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.291966 (+ 532us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:06.317446 9442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030fcc1b19c3489ea0ca320576ed54ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.317871 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:06.333781 9601 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.321025 9592 tablet_service.cc:1511] Processing CreateTablet for tablet 8650e2ab26ac47f8a5a6cd4dd6c30d50 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260503 13:58:06.306716 9587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d04573f959de406a9590a1603f633e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.332937 9566 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:06.324994 9591 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43746 (request call id 42) took 1041 ms. Trace:
I20260503 13:58:06.335492 9665 raft_consensus.cc:2804] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:06.335875 9665 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:06.336181 9665 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.338758 9453 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:06.339915 9492 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:06.334985 9591 rpcz_store.cc:276] 0503 13:58:05.283476 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:05.941289 (+657813us) service_pool.cc:224] Handling call
0503 13:58:05.942934 (+ 1645us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:05.943039 (+ 105us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.139133 (+196094us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.324538 (+185405us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.324943 (+ 405us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:06.335209 9567 leader_election.cc:343] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.348825 9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d04573f959de406a9590a1603f633e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.349404 9435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c55d864e0b0416ebdc09ce6e0310a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.316926 9651 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:06.326414 9443 rpcz_store.cc:276] 0503 13:58:05.304803 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.218117 (+913314us) service_pool.cc:224] Handling call
0503 13:58:06.222801 (+ 4684us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.222993 (+ 192us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.299449 (+ 76456us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.322676 (+ 23227us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.326011 (+ 3335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1190656}
I20260503 13:58:06.352315 9651 raft_consensus.cc:2804] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:06.354262 9651 raft_consensus.cc:697] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:06.355091 9651 consensus_queue.cc:237] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.359282 9491 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:06.309343 9439 tablet_service.cc:1511] Processing CreateTablet for tablet 754ac001a92e4dd6a0b6620803b83722 (DEFAULT_TABLE table=test-workload [id=54209d96daad4531bdc33057c81f7430]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260503 13:58:06.367753 9590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030fcc1b19c3489ea0ca320576ed54ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.372888 9592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8650e2ab26ac47f8a5a6cd4dd6c30d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.372272 9350 catalog_manager.cc:5671] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:06.373806 9601 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:06.378738 9665 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.387319 9665 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:06.388782 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:06.389420 9453 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.394225 9643 tablet_bootstrap.cc:492] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:06.394680 9643 ts_tablet_manager.cc:1403] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.119s user 0.015s sys 0.020s
I20260503 13:58:06.395573 9437 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46706 (request call id 51) took 1037 ms. Trace:
I20260503 13:58:06.395952 9437 rpcz_store.cc:276] 0503 13:58:05.357927 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.231460 (+873533us) service_pool.cc:224] Handling call
0503 13:58:06.303819 (+ 72359us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.304643 (+ 824us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.344724 (+ 40081us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.395119 (+ 50395us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.395513 (+ 394us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35072}
I20260503 13:58:06.396696 9439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 754ac001a92e4dd6a0b6620803b83722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260503 13:58:06.398633 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:06.399673 9441 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46706 (request call id 50) took 1046 ms. Trace:
I20260503 13:58:06.382380 9434 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46706 (request call id 49) took 1050 ms. Trace:
I20260503 13:58:06.400041 9441 rpcz_store.cc:276] 0503 13:58:05.353369 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.221972 (+868603us) service_pool.cc:224] Handling call
0503 13:58:06.238810 (+ 16838us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.238897 (+ 87us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.360654 (+121757us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.399216 (+ 38562us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.399616 (+ 400us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:06.400779 9567 leader_election.cc:343] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.402299 9453 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:06.403481 9566 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:06.400781 9434 rpcz_store.cc:276] 0503 13:58:05.331248 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.214768 (+883520us) service_pool.cc:224] Handling call
0503 13:58:06.242602 (+ 27834us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.242694 (+ 92us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.306220 (+ 63526us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.361505 (+ 55285us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.381681 (+ 20176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:06.404621 9665 raft_consensus.cc:2804] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:06.410470 9665 raft_consensus.cc:697] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:06.411404 9665 consensus_queue.cc:237] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.412143 9644 tablet_bootstrap.cc:654] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.408488 9651 raft_consensus.cc:2749] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260503 13:58:06.415117 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 56041us
I20260503 13:58:06.419940 9351 catalog_manager.cc:5671] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:06.420822 9643 raft_consensus.cc:359] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.421473 9643 raft_consensus.cc:385] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.421754 9642 tablet_bootstrap.cc:492] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:06.422240 9642 ts_tablet_manager.cc:1403] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.259s user 0.022s sys 0.006s
I20260503 13:58:06.425248 9642 raft_consensus.cc:359] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.425989 9642 raft_consensus.cc:385] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.426317 9642 raft_consensus.cc:740] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.427016 9642 consensus_queue.cc:260] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.421767 9643 raft_consensus.cc:740] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.433050 9643 consensus_queue.cc:260] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.437188 9642 ts_tablet_manager.cc:1434] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.015s user 0.004s sys 0.002s
I20260503 13:58:06.438185 9642 tablet_bootstrap.cc:492] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:06.445281 9643 ts_tablet_manager.cc:1434] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.050s user 0.008s sys 0.000s
I20260503 13:58:06.446067 9643 tablet_bootstrap.cc:492] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:06.451349 9643 tablet_bootstrap.cc:654] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.453052 9642 tablet_bootstrap.cc:654] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.460700 9644 tablet_bootstrap.cc:492] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:06.461140 9644 ts_tablet_manager.cc:1403] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.138s user 0.009s sys 0.013s
I20260503 13:58:06.462201 9642 tablet_bootstrap.cc:492] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:06.462685 9642 ts_tablet_manager.cc:1403] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.004s
I20260503 13:58:06.463941 9644 raft_consensus.cc:359] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.464515 9644 raft_consensus.cc:385] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.464825 9644 raft_consensus.cc:740] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.465435 9642 raft_consensus.cc:359] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.466159 9642 raft_consensus.cc:385] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.466444 9642 raft_consensus.cc:740] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.467141 9642 consensus_queue.cc:260] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.469389 9642 ts_tablet_manager.cc:1434] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260503 13:58:06.470309 9642 tablet_bootstrap.cc:492] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:06.470839 9643 tablet_bootstrap.cc:492] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:06.471266 9643 ts_tablet_manager.cc:1403] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.004s
I20260503 13:58:06.472908 9644 consensus_queue.cc:260] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.473994 9643 raft_consensus.cc:359] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.474668 9643 raft_consensus.cc:385] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.474951 9643 raft_consensus.cc:740] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.476094 9642 tablet_bootstrap.cc:654] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.475685 9643 consensus_queue.cc:260] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.478082 9643 ts_tablet_manager.cc:1434] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260503 13:58:06.479076 9643 tablet_bootstrap.cc:492] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:06.482527 9644 ts_tablet_manager.cc:1434] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.021s user 0.004s sys 0.001s
I20260503 13:58:06.485237 9643 tablet_bootstrap.cc:654] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.490024 9644 tablet_bootstrap.cc:492] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:06.496333 9644 tablet_bootstrap.cc:654] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.496997 9643 tablet_bootstrap.cc:492] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:06.497506 9643 ts_tablet_manager.cc:1403] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20260503 13:58:06.500401 9643 raft_consensus.cc:359] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.501008 9643 raft_consensus.cc:385] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.501206 9643 raft_consensus.cc:740] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.501947 9643 consensus_queue.cc:260] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.502622 9642 tablet_bootstrap.cc:492] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:06.503000 9644 tablet_bootstrap.cc:492] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:06.503063 9642 ts_tablet_manager.cc:1403] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.023s
I20260503 13:58:06.503695 9644 ts_tablet_manager.cc:1403] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260503 13:58:06.504231 9643 ts_tablet_manager.cc:1434] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260503 13:58:06.505203 9643 tablet_bootstrap.cc:492] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:06.506064 9642 raft_consensus.cc:359] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.506773 9642 raft_consensus.cc:385] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.507050 9642 raft_consensus.cc:740] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.507741 9644 raft_consensus.cc:359] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.508430 9644 raft_consensus.cc:385] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.508787 9644 raft_consensus.cc:740] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.509529 9644 consensus_queue.cc:260] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.516420 9643 tablet_bootstrap.cc:654] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.516492 9642 consensus_queue.cc:260] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.518152 9644 ts_tablet_manager.cc:1434] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20260503 13:58:06.518920 9642 ts_tablet_manager.cc:1434] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260503 13:58:06.519923 9642 tablet_bootstrap.cc:492] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:06.521391 9350 catalog_manager.cc:5671] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:06.526033 9642 tablet_bootstrap.cc:654] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.527889 9643 tablet_bootstrap.cc:492] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:06.528409 9643 ts_tablet_manager.cc:1403] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.011s
I20260503 13:58:06.528860 9644 tablet_bootstrap.cc:492] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:06.531402 9643 raft_consensus.cc:359] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.532193 9643 raft_consensus.cc:385] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.532523 9643 raft_consensus.cc:740] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.533391 9643 consensus_queue.cc:260] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.535043 9644 tablet_bootstrap.cc:654] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.535707 9643 ts_tablet_manager.cc:1434] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260503 13:58:06.536733 9643 tablet_bootstrap.cc:492] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:06.539376 9642 tablet_bootstrap.cc:492] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:06.539808 9642 ts_tablet_manager.cc:1403] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.004s
I20260503 13:58:06.542389 9642 raft_consensus.cc:359] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.543022 9642 raft_consensus.cc:385] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.543303 9642 raft_consensus.cc:740] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.544128 9643 tablet_bootstrap.cc:654] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.543979 9642 consensus_queue.cc:260] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.548801 9642 ts_tablet_manager.cc:1434] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20260503 13:58:06.549726 9642 tablet_bootstrap.cc:492] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:06.551088 9671 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.551537 9671 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.553565 9671 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:06.555491 9642 tablet_bootstrap.cc:654] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.555675 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:06.556250 9453 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:06.556945 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:06.562381 9491 leader_election.cc:343] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.562981 9643 tablet_bootstrap.cc:492] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:06.563318 9643 ts_tablet_manager.cc:1403] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.006s
I20260503 13:58:06.564136 9642 tablet_bootstrap.cc:492] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:06.564599 9642 ts_tablet_manager.cc:1403] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20260503 13:58:06.563376 9492 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:06.560039 9644 tablet_bootstrap.cc:492] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:06.565369 9643 raft_consensus.cc:359] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.565964 9643 raft_consensus.cc:385] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.566080 9644 ts_tablet_manager.cc:1403] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.037s user 0.020s sys 0.010s
I20260503 13:58:06.566248 9643 raft_consensus.cc:740] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.566859 9643 consensus_queue.cc:260] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.567212 9671 raft_consensus.cc:2804] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:06.567355 9642 raft_consensus.cc:359] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:06.568755 9643 ts_tablet_manager.cc:1434] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260503 13:58:06.574857 9642 raft_consensus.cc:385] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.567588 9671 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:06.575352 9642 raft_consensus.cc:740] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.569227 9644 raft_consensus.cc:359] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.575856 9643 tablet_bootstrap.cc:492] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:06.575971 9644 raft_consensus.cc:385] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.576378 9644 raft_consensus.cc:740] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.576146 9642 consensus_queue.cc:260] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:06.577145 9644 consensus_queue.cc:260] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.578526 9642 ts_tablet_manager.cc:1434] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260503 13:58:06.575428 9671 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.579459 9642 tablet_bootstrap.cc:492] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:06.581030 9643 tablet_bootstrap.cc:654] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.581748 9644 ts_tablet_manager.cc:1434] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.015s user 0.004s sys 0.007s
I20260503 13:58:06.582782 9644 tablet_bootstrap.cc:492] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:06.587240 9651 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.587693 9651 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.589550 9651 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:06.590883 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:06.591470 9453 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:06.593164 9492 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:06.593962 9651 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:06.594177 9644 tablet_bootstrap.cc:654] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.592373 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:06.595009 9642 tablet_bootstrap.cc:654] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.595009 9601 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:06.594301 9651 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:06.596815 9651 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.602768 9642 tablet_bootstrap.cc:492] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:06.603173 9642 ts_tablet_manager.cc:1403] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.016s
I20260503 13:58:06.605486 9642 raft_consensus.cc:359] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.606093 9642 raft_consensus.cc:385] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.606350 9642 raft_consensus.cc:740] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.607082 9642 consensus_queue.cc:260] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.610173 9642 ts_tablet_manager.cc:1434] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260503 13:58:06.610332 9644 tablet_bootstrap.cc:492] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:06.610934 9644 ts_tablet_manager.cc:1403] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.016s
I20260503 13:58:06.611040 9642 tablet_bootstrap.cc:492] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:06.613857 9644 raft_consensus.cc:359] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.614629 9644 raft_consensus.cc:385] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.614954 9644 raft_consensus.cc:740] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.615752 9644 consensus_queue.cc:260] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.616707 9642 tablet_bootstrap.cc:654] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.618237 9644 ts_tablet_manager.cc:1434] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260503 13:58:06.619284 9644 tablet_bootstrap.cc:492] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:06.604367 9651 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.621095 9651 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:06.621811 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:06.624891 9453 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.628042 9642 tablet_bootstrap.cc:492] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:06.628059 9644 tablet_bootstrap.cc:654] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.628702 9642 ts_tablet_manager.cc:1403] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
W20260503 13:58:06.630720 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:06.631189 9642 raft_consensus.cc:359] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.631858 9642 raft_consensus.cc:385] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.632108 9642 raft_consensus.cc:740] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.632869 9642 consensus_queue.cc:260] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.634878 9642 ts_tablet_manager.cc:1434] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260503 13:58:06.635648 9642 tablet_bootstrap.cc:492] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:06.640599 9642 tablet_bootstrap.cc:654] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.631156 9453 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:06.647433 9492 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:06.648308 9651 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:06.648794 9651 raft_consensus.cc:697] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:06.649627 9651 consensus_queue.cc:237] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.651418 9642 tablet_bootstrap.cc:492] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:06.651855 9642 ts_tablet_manager.cc:1403] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20260503 13:58:06.654191 9642 raft_consensus.cc:359] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.654839 9642 raft_consensus.cc:385] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.655097 9642 raft_consensus.cc:740] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.655803 9642 consensus_queue.cc:260] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.658128 9642 ts_tablet_manager.cc:1434] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260503 13:58:06.659016 9642 tablet_bootstrap.cc:492] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:06.624019 9671 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.622543 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:06.660723 9601 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.664038 9642 tablet_bootstrap.cc:654] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.667354 9601 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:06.674503 9642 tablet_bootstrap.cc:492] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:06.674932 9642 ts_tablet_manager.cc:1403] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.001s
I20260503 13:58:06.677345 9642 raft_consensus.cc:359] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.677961 9642 raft_consensus.cc:385] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.678226 9642 raft_consensus.cc:740] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.678925 9642 consensus_queue.cc:260] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.680755 9642 ts_tablet_manager.cc:1434] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260503 13:58:06.681565 9642 tablet_bootstrap.cc:492] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:06.686877 9642 tablet_bootstrap.cc:654] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.690472 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:06.691087 9453 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.697625 9671 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:06.698261 9642 tablet_bootstrap.cc:492] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:06.698668 9642 ts_tablet_manager.cc:1403] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.008s
I20260503 13:58:06.701102 9642 raft_consensus.cc:359] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.701390 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:06.701731 9642 raft_consensus.cc:385] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.702006 9642 raft_consensus.cc:740] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
W20260503 13:58:06.702535 9491 leader_election.cc:343] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.702584 9642 consensus_queue.cc:260] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.704538 9642 ts_tablet_manager.cc:1434] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260503 13:58:06.705353 9642 tablet_bootstrap.cc:492] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:06.710469 9642 tablet_bootstrap.cc:654] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.713459 9671 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.713954 9671 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.716773 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:06.719229 9492 leader_election.cc:343] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.721776 9665 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.722289 9665 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.726485 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:06.727149 9528 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:06.728992 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:06.728343 9567 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:06.731143 9649 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:06.731508 9649 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:06.731827 9649 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:58:06.736655 9566 leader_election.cc:343] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.724298 9665 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:06.740890 9649 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.743690 9453 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:06.744892 9492 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:06.746027 9651 raft_consensus.cc:2804] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:06.746496 9651 raft_consensus.cc:697] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:06.747318 9651 consensus_queue.cc:237] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.753159 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:06.754483 9566 leader_election.cc:343] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.756457 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:06.757680 9491 leader_election.cc:343] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.758143 9491 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:06.759143 9669 raft_consensus.cc:2749] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:06.764788 9671 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:06.767154 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:06.767750 9528 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.774732 9528 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:06.775843 9567 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:06.776738 9665 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:06.784482 9649 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:06.788599 9643 tablet_bootstrap.cc:492] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:06.789021 9643 ts_tablet_manager.cc:1403] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.213s user 0.083s sys 0.120s
I20260503 13:58:06.791498 9643 raft_consensus.cc:359] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.792104 9643 raft_consensus.cc:385] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.792302 9644 tablet_bootstrap.cc:492] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:06.792357 9643 raft_consensus.cc:740] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.793002 9644 ts_tablet_manager.cc:1403] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.174s user 0.050s sys 0.110s
I20260503 13:58:06.793154 9669 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.793337 9643 consensus_queue.cc:260] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.793660 9669 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:06.795710 9644 raft_consensus.cc:359] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.795198 9643 ts_tablet_manager.cc:1434] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260503 13:58:06.796343 9644 raft_consensus.cc:385] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.796765 9644 raft_consensus.cc:740] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.797137 9643 tablet_bootstrap.cc:492] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:06.797477 9669 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:06.797574 9644 consensus_queue.cc:260] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.800390 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:06.802536 9643 tablet_bootstrap.cc:654] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.802676 9644 ts_tablet_manager.cc:1434] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260503 13:58:06.803778 9644 tablet_bootstrap.cc:492] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:06.805359 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:06.802511 9492 leader_election.cc:343] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:06.807389 9491 leader_election.cc:343] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.807835 9491 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:06.809034 9669 consensus_queue.cc:1048] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:06.810096 9644 tablet_bootstrap.cc:654] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.810439 9669 raft_consensus.cc:2749] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:06.828234 9644 tablet_bootstrap.cc:492] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:06.828778 9644 ts_tablet_manager.cc:1403] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.025s user 0.021s sys 0.003s
I20260503 13:58:06.829560 9665 raft_consensus.cc:697] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:06.830480 9665 consensus_queue.cc:237] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.831569 9644 raft_consensus.cc:359] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.832276 9644 raft_consensus.cc:385] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.832602 9644 raft_consensus.cc:740] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.833496 9644 consensus_queue.cc:260] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.835743 9644 ts_tablet_manager.cc:1434] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260503 13:58:06.836673 9644 tablet_bootstrap.cc:492] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:06.842258 9644 tablet_bootstrap.cc:654] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.852517 9528 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.853585 9644 tablet_bootstrap.cc:492] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:06.854242 9644 ts_tablet_manager.cc:1403] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.011s
I20260503 13:58:06.856715 9644 raft_consensus.cc:359] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.857497 9644 raft_consensus.cc:385] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.857879 9644 raft_consensus.cc:740] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.858641 9644 consensus_queue.cc:260] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.862483 9644 ts_tablet_manager.cc:1434] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260503 13:58:06.866882 9651 consensus_queue.cc:1048] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:06.867447 9644 tablet_bootstrap.cc:492] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:06.869489 9649 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.869926 9649 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.871814 9649 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:06.873716 9650 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.873850 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:06.874208 9650 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:06.875221 9567 leader_election.cc:343] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.876104 9650 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:06.876749 9649 consensus_queue.cc:1048] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:06.878031 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:06.878360 9669 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.878808 9669 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.879983 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:06.880717 9527 raft_consensus.cc:2393] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:06.875993 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:06.881217 9453 raft_consensus.cc:2468] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:06.882056 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:06.882195 9566 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:06.882532 9642 tablet_bootstrap.cc:492] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:06.882622 9602 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:06.883029 9642 ts_tablet_manager.cc:1403] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.178s user 0.063s sys 0.108s
I20260503 13:58:06.883213 9649 raft_consensus.cc:2804] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:06.883608 9649 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:06.884050 9418 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:06.883973 9649 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.885457 9491 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:06.886123 9644 tablet_bootstrap.cc:654] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.886227 9650 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.886289 9708 raft_consensus.cc:2804] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:06.886883 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:06.887019 9708 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:06.887425 9708 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.887495 9452 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:06.891839 9701 consensus_queue.cc:1048] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:06.893602 9650 raft_consensus.cc:2749] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:06.893826 9708 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.896389 9708 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:06.885818 9642 raft_consensus.cc:359] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.897142 9643 tablet_bootstrap.cc:492] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:06.897326 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:06.897580 9643 ts_tablet_manager.cc:1403] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.101s user 0.039s sys 0.034s
I20260503 13:58:06.897859 9602 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.897506 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:06.900185 9453 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:06.901830 9649 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.903657 9649 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:06.904074 9649 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.880599 9669 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:06.906462 9453 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:06.897114 9642 raft_consensus.cc:385] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.905582 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:06.907349 9642 raft_consensus.cc:740] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.907948 9642 consensus_queue.cc:260] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.908270 9492 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:06.909440 9669 raft_consensus.cc:2804] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:06.909852 9642 ts_tablet_manager.cc:1434] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20260503 13:58:06.910929 9642 tablet_bootstrap.cc:492] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:06.911670 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:06.912225 9452 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:58:06.913196 9567 leader_election.cc:343] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.916447 9642 tablet_bootstrap.cc:654] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:06.909869 9669 raft_consensus.cc:697] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:06.919060 9669 consensus_queue.cc:237] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.900202 9643 raft_consensus.cc:359] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.921237 9643 raft_consensus.cc:385] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.921494 9643 raft_consensus.cc:740] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.922080 9643 consensus_queue.cc:260] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.916123 9350 catalog_manager.cc:5671] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:06.926836 9452 raft_consensus.cc:2468] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:06.904441 9649 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.930307 9649 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:06.931181 9712 consensus_queue.cc:1048] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260503 13:58:06.932410 9712 consensus_queue.cc:1048] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:06.944772 9719 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.945278 9719 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.947184 9719 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:06.950021 9650 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.950599 9650 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.951566 9602 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:06.952760 9650 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:06.959893 9644 tablet_bootstrap.cc:492] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:06.960331 9644 ts_tablet_manager.cc:1403] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.093s user 0.013s sys 0.013s
I20260503 13:58:06.960884 9643 ts_tablet_manager.cc:1434] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.063s user 0.016s sys 0.000s
I20260503 13:58:06.962967 9643 tablet_bootstrap.cc:492] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
W20260503 13:58:06.972747 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 53174.9us
I20260503 13:58:06.973397 9644 raft_consensus.cc:359] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.974135 9644 raft_consensus.cc:385] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:06.974396 9644 raft_consensus.cc:740] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.974572 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:06.981168 9644 consensus_queue.cc:260] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.971167 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:06.970796 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:06.989560 9644 ts_tablet_manager.cc:1434] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.029s user 0.005s sys 0.001s
I20260503 13:58:06.984669 9671 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.994704 9644 tablet_bootstrap.cc:492] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:06.990837 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:06.994546 9416 leader_election.cc:343] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:06.995424 9650 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.995404 9525 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:06.988690 9651 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:06.995975 9650 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:06.996637 9642 tablet_bootstrap.cc:492] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:07.157819 9708 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:07.206182 9642 ts_tablet_manager.cc:1403] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.295s user 0.032s sys 0.097s
I20260503 13:58:07.206473 9708 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.208523 9708 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.209030 9642 raft_consensus.cc:359] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.209666 9642 raft_consensus.cc:385] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.209530 9708 consensus_queue.cc:1048] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:07.209975 9642 raft_consensus.cc:740] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:06.985241 9526 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:07.210775 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:07.210559 9642 consensus_queue.cc:260] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.212080 9642 ts_tablet_manager.cc:1434] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260503 13:58:07.212944 9642 tablet_bootstrap.cc:492] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
W20260503 13:58:07.214457 9492 leader_election.cc:343] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.215538 9643 tablet_bootstrap.cc:654] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.077939 9649 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:07.216959 9649 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.218631 9649 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:07.219380 9567 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
W20260503 13:58:07.219892 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 151950us
I20260503 13:58:07.220005 9649 raft_consensus.cc:2804] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:07.220405 9649 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:07.220821 9649 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:58:07.221336 9721 log.cc:927] Time spent T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.227s user 0.001s sys 0.000s
W20260503 13:58:06.990543 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 70876.4us
I20260503 13:58:07.107486 9669 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:06.996713 9671 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.237215 9723 consensus_queue.cc:1048] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260503 13:58:07.239068 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:07.239645 9453 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:07.239596 9723 consensus_queue.cc:1048] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:07.242386 9566 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:07.233723 9651 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.235944 9669 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.244138 9717 raft_consensus.cc:2804] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:07.246299 9527 raft_consensus.cc:2468] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:07.247098 9566 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:07.247437 9671 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.247943 9712 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:07.248303 9712 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:07.248669 9712 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.257851 9669 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.253684 9717 raft_consensus.cc:697] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
W20260503 13:58:07.249065 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 59359.8us
I20260503 13:58:07.254731 9650 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.264457 9418 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:07.265558 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:07.263222 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:07.266319 9649 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.259609 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:07.270406 9650 raft_consensus.cc:2804] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:07.270131 9717 consensus_queue.cc:237] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.270769 9351 catalog_manager.cc:5671] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:07.271433 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:07.273401 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:07.274163 9644 tablet_bootstrap.cc:654] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.267338 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:07.275490 9651 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.276017 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:07.270780 9650 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260503 13:58:07.260864 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 45396.4us
W20260503 13:58:07.267565 9416 leader_election.cc:343] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.282122 9450 raft_consensus.cc:2393] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
W20260503 13:58:07.282028 9566 leader_election.cc:343] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.282320 9712 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.284993 9602 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.286751 9650 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.291105 9649 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:07.293365 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:07.313694 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 52383.6us
I20260503 13:58:07.315065 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:07.317201 9527 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:58:07.317076 9418 leader_election.cc:343] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.318022 9418 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:07.319372 9727 raft_consensus.cc:2749] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:07.320197 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:07.313553 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:07.321893 9712 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:07.322930 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:07.323465 9528 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.323191 9650 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.324307 9450 raft_consensus.cc:2393] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:07.326336 9642 tablet_bootstrap.cc:654] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.327540 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:07.329567 9416 leader_election.cc:343] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:07.328769 9491 leader_election.cc:343] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.329869 9650 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.337245 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:07.337787 9526 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.343297 9351 catalog_manager.cc:5671] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:07.345479 9351 catalog_manager.cc:5671] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:07.352706 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:07.333999 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:07.354658 9566 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:07.355703 9717 raft_consensus.cc:2749] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:07.357219 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:07.369413 9566 leader_election.cc:343] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:07.370237 9492 leader_election.cc:343] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.335896 9491 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:07.371276 9708 raft_consensus.cc:2749] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260503 13:58:07.371402 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 52302.1us
W20260503 13:58:07.372092 9491 leader_election.cc:343] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.372452 9491 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:07.373459 9491 leader_election.cc:343] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.373807 9491 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:07.374830 9491 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:07.379920 9526 raft_consensus.cc:2468] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
W20260503 13:58:07.383347 9566 consensus_peers.cc:597] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 -> Peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Couldn't send request to peer 1cb63886040047559d3893923cc9aaf6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260503 13:58:07.413081 9651 raft_consensus.cc:2749] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:07.413656 9669 consensus_queue.cc:1048] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:07.421000 9418 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:07.422261 9728 raft_consensus.cc:2804] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:07.422710 9728 raft_consensus.cc:697] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:07.423496 9728 consensus_queue.cc:237] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.390343 9528 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:07.393821 9527 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:07.399169 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:07.425663 9445 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.426664 9567 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:07.427534 9567 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:07.428361 9644 tablet_bootstrap.cc:492] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:07.407619 9723 raft_consensus.cc:2749] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:07.408162 9724 raft_consensus.cc:2749] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:07.428646 9649 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:07.429143 9649 raft_consensus.cc:697] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:07.429245 9712 raft_consensus.cc:2804] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:07.429425 9644 ts_tablet_manager.cc:1403] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.435s user 0.032s sys 0.065s
I20260503 13:58:07.429733 9712 raft_consensus.cc:697] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:07.429903 9649 consensus_queue.cc:237] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.431850 9644 raft_consensus.cc:359] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:07.431808 9712 consensus_queue.cc:237] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.432552 9644 raft_consensus.cc:385] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.432914 9644 raft_consensus.cc:740] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.433604 9644 consensus_queue.cc:260] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:07.435787 9644 ts_tablet_manager.cc:1434] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260503 13:58:07.436681 9644 tablet_bootstrap.cc:492] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:07.442358 9644 tablet_bootstrap.cc:654] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.451279 9445 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:07.455915 9350 catalog_manager.cc:5671] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
W20260503 13:58:07.470268 9731 log.cc:927] Time spent T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20260503 13:58:07.472774 9642 tablet_bootstrap.cc:492] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:07.478758 9642 ts_tablet_manager.cc:1403] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.266s user 0.042s sys 0.049s
I20260503 13:58:07.481518 9723 consensus_queue.cc:1048] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:07.482888 9649 consensus_queue.cc:1048] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:07.486277 9642 raft_consensus.cc:359] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.486936 9642 raft_consensus.cc:385] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.487192 9642 raft_consensus.cc:740] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.487876 9642 consensus_queue.cc:260] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.488534 9723 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:07.488987 9723 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.489842 9642 ts_tablet_manager.cc:1434] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260503 13:58:07.490721 9642 tablet_bootstrap.cc:492] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:07.491880 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:07.498996 9643 tablet_bootstrap.cc:492] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:07.499447 9643 ts_tablet_manager.cc:1403] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.537s user 0.042s sys 0.032s
I20260503 13:58:07.501211 9650 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:07.501641 9650 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:07.503993 9650 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.499526 9723 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.504968 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:07.505945 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:07.506834 9527 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:07.507400 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:07.508179 9642 tablet_bootstrap.cc:654] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
W20260503 13:58:07.500841 9492 leader_election.cc:343] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.507984 9418 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:07.510416 9644 tablet_bootstrap.cc:492] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
W20260503 13:58:07.509831 9491 leader_election.cc:343] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.510951 9644 ts_tablet_manager.cc:1403] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.074s user 0.035s sys 0.020s
I20260503 13:58:07.511334 9491 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:07.513432 9723 raft_consensus.cc:2749] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:07.511729 9728 raft_consensus.cc:2804] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:07.515923 9728 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:07.516264 9728 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.516083 9350 catalog_manager.cc:5671] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 1 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:07.528774 9644 raft_consensus.cc:359] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.529439 9644 raft_consensus.cc:385] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.529695 9644 raft_consensus.cc:740] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.530267 9644 consensus_queue.cc:260] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:07.533537 9416 leader_election.cc:343] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.533767 9723 consensus_queue.cc:1048] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:07.537370 9643 raft_consensus.cc:359] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:07.538003 9643 raft_consensus.cc:385] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.538254 9643 raft_consensus.cc:740] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.538831 9643 consensus_queue.cc:260] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:07.540787 9643 ts_tablet_manager.cc:1434] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20260503 13:58:07.541626 9643 tablet_bootstrap.cc:492] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:07.544329 9349 catalog_manager.cc:5671] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:07.547497 9643 tablet_bootstrap.cc:654] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.547055 9349 catalog_manager.cc:5671] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:07.549530 9644 ts_tablet_manager.cc:1434] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20260503 13:58:07.550279 9644 tablet_bootstrap.cc:492] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:07.553954 9642 tablet_bootstrap.cc:492] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:07.554554 9642 ts_tablet_manager.cc:1403] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.064s user 0.011s sys 0.000s
I20260503 13:58:07.556763 9728 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:07.556877 9642 raft_consensus.cc:359] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.557677 9642 raft_consensus.cc:385] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.557945 9642 raft_consensus.cc:740] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.558458 9642 consensus_queue.cc:260] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.562373 9643 tablet_bootstrap.cc:492] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:07.562810 9643 ts_tablet_manager.cc:1403] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.008s
I20260503 13:58:07.563593 9642 ts_tablet_manager.cc:1434] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20260503 13:58:07.563999 9644 tablet_bootstrap.cc:654] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.564425 9642 tablet_bootstrap.cc:492] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:07.565521 9643 raft_consensus.cc:359] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.565897 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:07.566190 9643 raft_consensus.cc:385] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.566462 9528 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.566581 9643 raft_consensus.cc:740] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.567350 9643 consensus_queue.cc:260] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.569602 9643 ts_tablet_manager.cc:1434] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260503 13:58:07.570474 9642 tablet_bootstrap.cc:654] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.570500 9643 tablet_bootstrap.cc:492] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:07.573696 9528 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:07.574793 9418 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:07.575628 9727 raft_consensus.cc:2804] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:07.576076 9727 raft_consensus.cc:697] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:07.577096 9643 tablet_bootstrap.cc:654] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.577050 9727 consensus_queue.cc:237] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:07.582022 9728 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.585007 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:07.585569 9601 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.590435 9643 tablet_bootstrap.cc:492] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:07.590847 9643 ts_tablet_manager.cc:1403] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.003s
I20260503 13:58:07.595780 9601 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:07.596092 9642 tablet_bootstrap.cc:492] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:07.596602 9642 ts_tablet_manager.cc:1403] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.007s
I20260503 13:58:07.598146 9351 catalog_manager.cc:5671] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 1 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:07.599745 9642 raft_consensus.cc:359] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.600337 9642 raft_consensus.cc:385] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.600483 9644 tablet_bootstrap.cc:492] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:07.600616 9642 raft_consensus.cc:740] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.600905 9644 ts_tablet_manager.cc:1403] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.003s
I20260503 13:58:07.601202 9642 consensus_queue.cc:260] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.598203 9643 raft_consensus.cc:359] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.602643 9643 raft_consensus.cc:385] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.602908 9643 raft_consensus.cc:740] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.603253 9644 raft_consensus.cc:359] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.603449 9643 consensus_queue.cc:260] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.603848 9644 raft_consensus.cc:385] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.604182 9642 ts_tablet_manager.cc:1434] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260503 13:58:07.605157 9642 tablet_bootstrap.cc:492] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:07.605376 9643 ts_tablet_manager.cc:1434] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.014s user 0.003s sys 0.004s
I20260503 13:58:07.606365 9643 tablet_bootstrap.cc:492] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:07.610347 9642 tablet_bootstrap.cc:654] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.611788 9643 tablet_bootstrap.cc:654] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.604182 9644 raft_consensus.cc:740] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.615170 9644 consensus_queue.cc:260] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.617482 9644 ts_tablet_manager.cc:1434] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.016s user 0.004s sys 0.002s
I20260503 13:58:07.630511 9644 tablet_bootstrap.cc:492] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:07.632210 9643 tablet_bootstrap.cc:492] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:07.632804 9643 ts_tablet_manager.cc:1403] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.012s
I20260503 13:58:07.635151 9643 raft_consensus.cc:359] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.635852 9643 raft_consensus.cc:385] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.636161 9643 raft_consensus.cc:740] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.637058 9644 tablet_bootstrap.cc:654] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.636907 9643 consensus_queue.cc:260] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.642656 9643 ts_tablet_manager.cc:1434] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20260503 13:58:07.643597 9643 tablet_bootstrap.cc:492] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
W20260503 13:58:07.646202 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:07.648149 9642 tablet_bootstrap.cc:492] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:07.648705 9642 ts_tablet_manager.cc:1403] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.044s user 0.018s sys 0.013s
I20260503 13:58:07.652101 9642 raft_consensus.cc:359] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.652415 9643 tablet_bootstrap.cc:654] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.652784 9642 raft_consensus.cc:385] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.653210 9642 raft_consensus.cc:740] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.653484 9644 tablet_bootstrap.cc:492] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:07.653899 9644 ts_tablet_manager.cc:1403] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.002s
I20260503 13:58:07.653720 9642 consensus_queue.cc:260] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.655741 9642 ts_tablet_manager.cc:1434] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260503 13:58:07.656971 9642 tablet_bootstrap.cc:492] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:07.657888 9644 raft_consensus.cc:359] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.659189 9644 raft_consensus.cc:385] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.659439 9644 raft_consensus.cc:740] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.660157 9644 consensus_queue.cc:260] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.663183 9642 tablet_bootstrap.cc:654] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.665367 9644 ts_tablet_manager.cc:1434] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20260503 13:58:07.666266 9644 tablet_bootstrap.cc:492] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:07.671519 9644 tablet_bootstrap.cc:654] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.676082 9643 tablet_bootstrap.cc:492] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:07.676371 9642 tablet_bootstrap.cc:492] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:07.676482 9643 ts_tablet_manager.cc:1403] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.007s
I20260503 13:58:07.676755 9642 ts_tablet_manager.cc:1403] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20260503 13:58:07.679181 9642 raft_consensus.cc:359] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.679051 9643 raft_consensus.cc:359] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.679693 9642 raft_consensus.cc:385] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.679780 9643 raft_consensus.cc:385] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.680034 9642 raft_consensus.cc:740] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.680094 9643 raft_consensus.cc:740] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.681057 9644 tablet_bootstrap.cc:492] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:07.680840 9643 consensus_queue.cc:260] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.680728 9642 consensus_queue.cc:260] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.681463 9644 ts_tablet_manager.cc:1403] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260503 13:58:07.682653 9642 ts_tablet_manager.cc:1434] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260503 13:58:07.682653 9643 ts_tablet_manager.cc:1434] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260503 13:58:07.683595 9642 tablet_bootstrap.cc:492] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:07.683954 9643 tablet_bootstrap.cc:492] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:07.684340 9644 raft_consensus.cc:359] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.684986 9644 raft_consensus.cc:385] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.685243 9644 raft_consensus.cc:740] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.685902 9644 consensus_queue.cc:260] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.687054 9727 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:07.687455 9727 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.689347 9642 tablet_bootstrap.cc:654] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.690919 9644 ts_tablet_manager.cc:1434] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260503 13:58:07.691766 9644 tablet_bootstrap.cc:492] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:07.691655 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:07.692981 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:07.693749 9528 raft_consensus.cc:2393] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:07.695549 9416 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:07.697026 9728 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.697193 9644 tablet_bootstrap.cc:654] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.701160 9717 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:07.701612 9717 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.703539 9717 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:07.705470 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:07.707947 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:07.708724 9528 raft_consensus.cc:2393] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:07.709872 9567 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:07.711081 9727 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.711580 9717 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.721415 9644 tablet_bootstrap.cc:492] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:07.721779 9644 ts_tablet_manager.cc:1403] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.020s
I20260503 13:58:07.722337 9642 tablet_bootstrap.cc:492] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:07.722808 9642 ts_tablet_manager.cc:1403] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.039s user 0.017s sys 0.020s
I20260503 13:58:07.724622 9728 raft_consensus.cc:2749] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:07.724349 9644 raft_consensus.cc:359] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.725072 9644 raft_consensus.cc:385] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.725334 9644 raft_consensus.cc:740] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.725242 9642 raft_consensus.cc:359] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.725943 9642 raft_consensus.cc:385] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.726272 9642 raft_consensus.cc:740] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.726052 9644 consensus_queue.cc:260] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.726851 9642 consensus_queue.cc:260] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.727670 9644 ts_tablet_manager.cc:1434] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260503 13:58:07.727636 9727 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:07.728469 9642 ts_tablet_manager.cc:1434] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260503 13:58:07.728720 9644 tablet_bootstrap.cc:492] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:07.728660 9727 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.729291 9642 tablet_bootstrap.cc:492] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:07.731259 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:07.731834 9528 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:07.732833 9418 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:07.733588 9650 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:07.733945 9650 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:07.734249 9650 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.735638 9642 tablet_bootstrap.cc:654] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.735625 9644 tablet_bootstrap.cc:654] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.742130 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:07.743479 9416 leader_election.cc:343] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.744072 9727 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.744966 9650 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.748247 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:07.748860 9528 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.753654 9644 tablet_bootstrap.cc:492] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:07.754006 9644 ts_tablet_manager.cc:1403] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.007s
I20260503 13:58:07.755510 9528 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:07.756336 9644 raft_consensus.cc:359] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.756960 9644 raft_consensus.cc:385] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.757230 9644 raft_consensus.cc:740] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.757084 9418 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:07.757869 9644 consensus_queue.cc:260] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.758330 9727 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:07.758834 9727 raft_consensus.cc:697] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:07.759653 9727 consensus_queue.cc:237] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.760217 9644 ts_tablet_manager.cc:1434] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260503 13:58:07.761070 9644 tablet_bootstrap.cc:492] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:07.762603 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:07.762745 9650 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:07.763876 9416 leader_election.cc:343] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.762378 9717 raft_consensus.cc:2749] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:07.765748 9643 tablet_bootstrap.cc:654] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.766258 9644 tablet_bootstrap.cc:654] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.772910 9644 tablet_bootstrap.cc:492] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:07.773324 9644 ts_tablet_manager.cc:1403] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260503 13:58:07.776324 9644 raft_consensus.cc:359] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.776980 9644 raft_consensus.cc:385] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.777226 9644 raft_consensus.cc:740] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.778825 9644 consensus_queue.cc:260] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.780377 9717 consensus_queue.cc:1048] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:07.780751 9644 ts_tablet_manager.cc:1434] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260503 13:58:07.781562 9644 tablet_bootstrap.cc:492] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:07.782224 9725 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:07.782667 9725 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.786476 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:07.786649 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:07.787163 9601 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:07.787364 9644 tablet_bootstrap.cc:654] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.787081 9449 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:07.788683 9725 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.780828 9349 catalog_manager.cc:5671] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 1 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:07.790050 9491 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:07.791193 9725 raft_consensus.cc:2804] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:07.791553 9725 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:07.791872 9725 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.800915 9643 tablet_bootstrap.cc:492] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:07.803020 9643 ts_tablet_manager.cc:1403] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.119s user 0.008s sys 0.010s
I20260503 13:58:07.807925 9643 raft_consensus.cc:359] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.808598 9643 raft_consensus.cc:385] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.808868 9643 raft_consensus.cc:740] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.809475 9643 consensus_queue.cc:260] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.811714 9725 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.815122 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:07.815451 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:07.815621 9601 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.816828 9643 ts_tablet_manager.cc:1434] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260503 13:58:07.815989 9445 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.818727 9643 tablet_bootstrap.cc:492] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:07.830521 9528 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.817476 9725 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.835510 9643 tablet_bootstrap.cc:654] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.836639 9717 consensus_queue.cc:1048] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260503 13:58:07.844210 9717 consensus_queue.cc:1048] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:07.840220 9601 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:07.849835 9642 tablet_bootstrap.cc:492] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:07.850406 9642 ts_tablet_manager.cc:1403] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.121s user 0.048s sys 0.049s
I20260503 13:58:07.850770 9491 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:07.849890 9445 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:07.851663 9723 raft_consensus.cc:2804] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:07.852170 9723 raft_consensus.cc:697] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:07.853048 9723 consensus_queue.cc:237] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.854646 9642 raft_consensus.cc:359] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.855295 9642 raft_consensus.cc:385] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.855548 9642 raft_consensus.cc:740] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.856148 9642 consensus_queue.cc:260] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.859048 9642 ts_tablet_manager.cc:1434] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260503 13:58:07.861294 9642 tablet_bootstrap.cc:492] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:07.873413 9727 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:07.873912 9727 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.875794 9727 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.877823 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:07.878384 9601 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:07.881542 9714 consensus_queue.cc:1048] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260503 13:58:07.882505 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:07.883085 9528 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:07.885401 9416 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:07.887889 9650 raft_consensus.cc:2804] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:07.888252 9650 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:07.888585 9650 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.893198 9727 consensus_queue.cc:1048] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:07.899166 9650 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.904234 9649 consensus_queue.cc:1048] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:07.905049 9642 tablet_bootstrap.cc:654] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.907510 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:07.907552 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:07.908283 9601 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.908588 9525 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.915498 9714 consensus_queue.cc:1048] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:07.916181 9671 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:07.916625 9671 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.921989 9650 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.931813 9644 tablet_bootstrap.cc:492] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:07.932252 9644 ts_tablet_manager.cc:1403] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.151s user 0.031s sys 0.032s
I20260503 13:58:07.932262 9643 tablet_bootstrap.cc:492] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:07.933127 9643 ts_tablet_manager.cc:1403] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.115s user 0.026s sys 0.023s
I20260503 13:58:07.949859 9671 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.951787 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:07.952255 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:07.952802 9451 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
W20260503 13:58:07.953100 9491 leader_election.cc:343] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.953863 9492 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:07.955061 9649 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:07.955540 9649 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.957479 9649 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:07.958762 9671 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:07.960107 9650 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:07.960587 9650 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.962488 9650 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.960512 9671 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:07.963130 9671 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.951390 9643 raft_consensus.cc:359] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.964704 9643 raft_consensus.cc:385] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.964996 9643 raft_consensus.cc:740] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.965615 9643 consensus_queue.cc:260] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.965965 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:07.967464 9643 ts_tablet_manager.cc:1434] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.034s user 0.003s sys 0.002s
I20260503 13:58:07.968313 9643 tablet_bootstrap.cc:492] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:07.958591 9644 raft_consensus.cc:359] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.959594 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:07.968827 9669 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:07.968982 9644 raft_consensus.cc:385] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:07.948928 9601 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:07.959167 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:07.969267 9669 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.969267 9644 raft_consensus.cc:740] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:07.970718 9644 consensus_queue.cc:260] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:07.972054 9566 leader_election.cc:343] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.976209 9644 ts_tablet_manager.cc:1434] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.044s user 0.002s sys 0.003s
I20260503 13:58:07.971596 9525 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:07.977591 9644 tablet_bootstrap.cc:492] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:07.976585 9727 consensus_queue.cc:1048] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260503 13:58:07.977161 9416 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:07.977239 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:07.984279 9728 raft_consensus.cc:2804] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:07.986747 9728 raft_consensus.cc:697] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:07.987490 9728 consensus_queue.cc:237] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:07.989231 9643 tablet_bootstrap.cc:654] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:07.992102 9418 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:07.992619 9669 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:07.993068 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:07.993714 9451 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
W20260503 13:58:07.980672 9567 leader_election.cc:343] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:07.993047 9727 raft_consensus.cc:2749] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:07.996974 9492 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:08.000145 9725 raft_consensus.cc:2804] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:08.001390 9725 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:08.001708 9725 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:07.998355 9649 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:08.004925 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:08.004839 9649 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.007105 9649 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:07.998165 9567 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:08.008693 9649 raft_consensus.cc:2749] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:08.010609 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:08.011278 9527 raft_consensus.cc:2468] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:08.012301 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:08.015883 9566 leader_election.cc:343] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:08.016507 9491 leader_election.cc:343] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.017931 9725 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.020862 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:08.021374 9449 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.025493 9349 catalog_manager.cc:5671] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 1 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:08.025760 9351 catalog_manager.cc:5671] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:08.026538 9567 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:08.029989 9642 tablet_bootstrap.cc:492] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:08.030457 9642 ts_tablet_manager.cc:1403] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.169s user 0.017s sys 0.005s
I20260503 13:58:08.031395 9649 raft_consensus.cc:2804] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:08.031750 9649 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:08.032058 9649 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.038596 9449 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:08.039237 9725 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.043524 9492 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:08.044982 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:08.046931 9725 raft_consensus.cc:2804] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
W20260503 13:58:08.049116 9491 leader_election.cc:343] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.047930 9725 raft_consensus.cc:697] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:08.050674 9725 consensus_queue.cc:237] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.050473 9644 tablet_bootstrap.cc:654] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.048296 9649 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.054281 9728 consensus_queue.cc:1048] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:08.054272 9449 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.059487 9642 raft_consensus.cc:359] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.060335 9642 raft_consensus.cc:385] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.060668 9642 raft_consensus.cc:740] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.064836 9649 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:08.065508 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:08.066804 9566 leader_election.cc:343] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.068241 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:08.068030 9671 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.068796 9526 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.071285 9643 tablet_bootstrap.cc:492] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:08.073307 9643 ts_tablet_manager.cc:1403] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.105s user 0.014s sys 0.000s
I20260503 13:58:08.076848 9643 raft_consensus.cc:359] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.073717 9642 consensus_queue.cc:260] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.083096 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:08.083636 9451 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.084766 9642 ts_tablet_manager.cc:1434] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.054s user 0.006s sys 0.001s
I20260503 13:58:08.082026 9671 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.080214 9526 raft_consensus.cc:2468] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:08.083107 9643 raft_consensus.cc:385] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.086612 9643 raft_consensus.cc:740] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.086758 9644 tablet_bootstrap.cc:492] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:08.086607 9567 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:08.088160 9644 ts_tablet_manager.cc:1403] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.111s user 0.017s sys 0.009s
I20260503 13:58:08.088847 9714 raft_consensus.cc:2804] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:08.089298 9714 raft_consensus.cc:697] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:08.088492 9643 consensus_queue.cc:260] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.090309 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:08.090071 9714 consensus_queue.cc:237] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.090714 9644 raft_consensus.cc:359] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.091356 9644 raft_consensus.cc:385] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.091640 9644 raft_consensus.cc:740] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.092867 9643 ts_tablet_manager.cc:1434] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260503 13:58:08.093773 9643 tablet_bootstrap.cc:492] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
W20260503 13:58:08.093817 9491 leader_election.cc:343] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.085752 9642 tablet_bootstrap.cc:492] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:08.099828 9644 consensus_queue.cc:260] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.100584 9643 tablet_bootstrap.cc:654] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.102504 9727 consensus_queue.cc:1048] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:08.103705 9644 ts_tablet_manager.cc:1434] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.015s user 0.004s sys 0.001s
I20260503 13:58:08.103188 9349 catalog_manager.cc:5671] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:08.104542 9644 tablet_bootstrap.cc:492] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:08.105850 9642 tablet_bootstrap.cc:654] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.093262 9451 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:08.123296 9644 tablet_bootstrap.cc:654] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.123095 9729 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:08.129304 9729 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.137411 9492 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:08.139671 9669 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:08.140115 9669 raft_consensus.cc:697] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:08.140898 9669 consensus_queue.cc:237] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.148386 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:08.141367 9729 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.150784 9643 tablet_bootstrap.cc:492] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:08.151257 9643 ts_tablet_manager.cc:1403] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.058s user 0.022s sys 0.004s
I20260503 13:58:08.152305 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:08.153549 9416 leader_election.cc:343] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.154584 9526 raft_consensus.cc:2468] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:08.155979 9418 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:08.157088 9729 raft_consensus.cc:2804] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:08.157444 9729 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:08.157763 9729 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.163656 9350 catalog_manager.cc:5671] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:08.173522 9644 tablet_bootstrap.cc:492] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:08.188625 9729 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.191987 9644 ts_tablet_manager.cc:1403] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.088s user 0.007s sys 0.004s
I20260503 13:58:08.195441 9642 tablet_bootstrap.cc:492] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:08.195930 9642 ts_tablet_manager.cc:1403] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.110s user 0.017s sys 0.015s
I20260503 13:58:08.196902 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:08.197302 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:08.197445 9526 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:58:08.198514 9416 leader_election.cc:343] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.199054 9642 raft_consensus.cc:359] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.199731 9642 raft_consensus.cc:385] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.200037 9642 raft_consensus.cc:740] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.200898 9725 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:08.201303 9725 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.203049 9725 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.204378 9729 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.204515 9526 raft_consensus.cc:2468] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:08.205130 9643 raft_consensus.cc:359] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.205727 9643 raft_consensus.cc:385] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.205870 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:08.203018 9349 catalog_manager.cc:5671] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:08.206905 9671 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:08.207234 9492 leader_election.cc:343] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.207314 9671 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.205996 9643 raft_consensus.cc:740] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.209151 9671 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.209110 9643 consensus_queue.cc:260] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.209348 9642 consensus_queue.cc:260] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.211418 9643 ts_tablet_manager.cc:1434] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.060s user 0.003s sys 0.002s
I20260503 13:58:08.210994 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:08.212280 9643 tablet_bootstrap.cc:492] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
W20260503 13:58:08.212638 9491 leader_election.cc:343] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.214187 9642 ts_tablet_manager.cc:1434] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.018s user 0.004s sys 0.001s
I20260503 13:58:08.215094 9642 tablet_bootstrap.cc:492] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:08.216307 9418 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:08.217624 9727 raft_consensus.cc:2804] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:08.218072 9727 raft_consensus.cc:697] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:08.218750 9727 consensus_queue.cc:237] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.220472 9642 tablet_bootstrap.cc:654] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.221113 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:08.221709 9601 raft_consensus.cc:2468] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:08.222617 9643 tablet_bootstrap.cc:654] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.222708 9491 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:08.224658 9671 raft_consensus.cc:2804] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:08.224992 9671 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:08.225277 9671 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.231678 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:08.232250 9449 raft_consensus.cc:2468] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:08.233541 9669 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:08.234002 9669 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.236706 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:08.237875 9491 leader_election.cc:343] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.233541 9492 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:08.239609 9725 raft_consensus.cc:2804] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:08.240010 9643 tablet_bootstrap.cc:492] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:08.239948 9725 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:08.240341 9725 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.240617 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:08.242873 9644 raft_consensus.cc:359] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.243494 9644 raft_consensus.cc:385] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.243768 9644 raft_consensus.cc:740] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.244406 9644 consensus_queue.cc:260] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:08.245416 9492 leader_election.cc:343] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.245832 9492 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:08.246289 9725 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.246528 9669 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.246932 9724 raft_consensus.cc:2749] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:08.247140 9643 ts_tablet_manager.cc:1403] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.035s user 0.015s sys 0.003s
I20260503 13:58:08.248270 9351 catalog_manager.cc:5671] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 1 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:08.248708 9671 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.249133 9643 raft_consensus.cc:359] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.249758 9643 raft_consensus.cc:385] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.250020 9643 raft_consensus.cc:740] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.250114 9725 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.250620 9643 consensus_queue.cc:260] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.252317 9643 ts_tablet_manager.cc:1434] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260503 13:58:08.252441 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:08.253180 9643 tablet_bootstrap.cc:492] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:08.253319 9449 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.251075 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:08.258832 9643 tablet_bootstrap.cc:654] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
W20260503 13:58:08.258940 9491 leader_election.cc:343] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.263064 9644 ts_tablet_manager.cc:1434] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.069s user 0.005s sys 0.000s
I20260503 13:58:08.263413 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:08.264176 9644 tablet_bootstrap.cc:492] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
W20260503 13:58:08.264467 9492 leader_election.cc:343] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.266816 9671 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.267251 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:08.267530 9727 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:08.267957 9601 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.268026 9727 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.270056 9727 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.272375 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:08.273962 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:08.274361 9416 leader_election.cc:343] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:08.275123 9418 leader_election.cc:343] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.275503 9418 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:08.276458 9727 raft_consensus.cc:2749] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:08.283612 9449 raft_consensus.cc:2468] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:08.284183 9727 consensus_queue.cc:1048] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:08.288308 9643 tablet_bootstrap.cc:492] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:08.293426 9492 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:08.293723 9643 ts_tablet_manager.cc:1403] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.018s
I20260503 13:58:08.294531 9671 raft_consensus.cc:2804] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:08.294986 9671 raft_consensus.cc:697] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:08.295724 9671 consensus_queue.cc:237] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.288327 9601 raft_consensus.cc:2468] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:08.288928 9642 tablet_bootstrap.cc:492] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:08.291211 9644 tablet_bootstrap.cc:654] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.296696 9643 raft_consensus.cc:359] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.297461 9643 raft_consensus.cc:385] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.297863 9643 raft_consensus.cc:740] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.298611 9643 consensus_queue.cc:260] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.299172 9491 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:08.300323 9725 raft_consensus.cc:2804] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:08.300946 9725 raft_consensus.cc:697] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:08.301677 9725 consensus_queue.cc:237] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.297485 9642 ts_tablet_manager.cc:1403] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.083s user 0.012s sys 0.000s
I20260503 13:58:08.309435 9643 ts_tablet_manager.cc:1434] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.015s user 0.007s sys 0.001s
I20260503 13:58:08.310489 9601 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.317309 9643 tablet_bootstrap.cc:492] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:08.321967 9642 raft_consensus.cc:359] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.322544 9642 raft_consensus.cc:385] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.322757 9642 raft_consensus.cc:740] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.323208 9642 consensus_queue.cc:260] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.324494 9724 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:08.324920 9642 ts_tablet_manager.cc:1434] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260503 13:58:08.324950 9724 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.325843 9642 tablet_bootstrap.cc:492] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:08.326946 9724 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.328485 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:08.329113 9449 raft_consensus.cc:2468] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:08.330322 9492 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:08.332448 9642 tablet_bootstrap.cc:654] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.332621 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:08.334241 9724 raft_consensus.cc:2804] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260503 13:58:08.335533 9491 leader_election.cc:343] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.342317 9669 consensus_queue.cc:1048] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260503 13:58:08.340179 9724 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:08.348577 9724 raft_consensus.cc:3060] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.356850 9728 consensus_queue.cc:1048] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:08.356700 9349 catalog_manager.cc:5671] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:08.366003 9724 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.369728 9724 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.370709 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:08.372354 9642 tablet_bootstrap.cc:492] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
W20260503 13:58:08.372823 9491 leader_election.cc:343] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.372898 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:08.373503 9451 raft_consensus.cc:3060] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.376987 9642 ts_tablet_manager.cc:1403] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.015s
I20260503 13:58:08.381026 9643 tablet_bootstrap.cc:654] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.382076 9642 raft_consensus.cc:359] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.382786 9642 raft_consensus.cc:385] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.387542 9724 consensus_queue.cc:1048] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:08.387187 9351 catalog_manager.cc:5671] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:08.390093 9451 raft_consensus.cc:2468] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:08.391187 9492 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:08.392292 9669 raft_consensus.cc:2804] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:08.392768 9669 raft_consensus.cc:697] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:08.393512 9669 consensus_queue.cc:237] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.399170 9642 raft_consensus.cc:740] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.401439 9642 consensus_queue.cc:260] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.405910 9644 tablet_bootstrap.cc:492] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:08.409094 9644 ts_tablet_manager.cc:1403] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.145s user 0.018s sys 0.022s
I20260503 13:58:08.411981 9644 raft_consensus.cc:359] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.412721 9644 raft_consensus.cc:385] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.413035 9644 raft_consensus.cc:740] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.413698 9644 consensus_queue.cc:260] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.416371 9644 ts_tablet_manager.cc:1434] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260503 13:58:08.417342 9644 tablet_bootstrap.cc:492] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:08.418689 9349 catalog_manager.cc:5671] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:08.421442 9642 ts_tablet_manager.cc:1434] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20260503 13:58:08.422456 9642 tablet_bootstrap.cc:492] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:08.429219 9728 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:08.439640 9728 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.441022 9642 tablet_bootstrap.cc:654] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.443774 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:08.446588 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:08.447265 9601 raft_consensus.cc:2393] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:08.447506 9728 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.449278 9643 tablet_bootstrap.cc:492] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:08.449741 9643 ts_tablet_manager.cc:1403] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.133s user 0.025s sys 0.026s
I20260503 13:58:08.449820 9416 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:08.450876 9728 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.452059 9644 tablet_bootstrap.cc:654] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.454730 9643 raft_consensus.cc:359] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.455399 9643 raft_consensus.cc:385] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.455694 9643 raft_consensus.cc:740] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.456414 9643 consensus_queue.cc:260] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.458683 9643 ts_tablet_manager.cc:1434] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260503 13:58:08.459406 9643 tablet_bootstrap.cc:492] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:08.461468 9728 raft_consensus.cc:2749] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:08.462214 9644 tablet_bootstrap.cc:492] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:08.462682 9644 ts_tablet_manager.cc:1403] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.007s
I20260503 13:58:08.465719 9644 raft_consensus.cc:359] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.466287 9644 raft_consensus.cc:385] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.466482 9644 raft_consensus.cc:740] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.466913 9644 consensus_queue.cc:260] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.468905 9644 ts_tablet_manager.cc:1434] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260503 13:58:08.471028 9642 tablet_bootstrap.cc:492] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:08.471441 9642 ts_tablet_manager.cc:1403] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.002s
I20260503 13:58:08.472261 9644 tablet_bootstrap.cc:492] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:08.474304 9642 raft_consensus.cc:359] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.474900 9642 raft_consensus.cc:385] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.475143 9642 raft_consensus.cc:740] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.475764 9642 consensus_queue.cc:260] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.476624 9729 consensus_queue.cc:1048] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:08.478608 9644 tablet_bootstrap.cc:654] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.471555 9643 tablet_bootstrap.cc:654] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.484918 9642 ts_tablet_manager.cc:1434] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260503 13:58:08.487293 9642 tablet_bootstrap.cc:492] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:08.492997 9642 tablet_bootstrap.cc:654] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.496922 9669 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:08.497400 9669 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.499135 9643 tablet_bootstrap.cc:492] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:08.499188 9669 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.499701 9643 ts_tablet_manager.cc:1403] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.003s
I20260503 13:58:08.502019 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:08.502597 9600 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:08.503473 9643 raft_consensus.cc:359] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.503794 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:08.504966 9643 raft_consensus.cc:385] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.505003 9491 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:08.505373 9643 raft_consensus.cc:740] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
W20260503 13:58:08.505632 9492 leader_election.cc:343] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.506040 9643 consensus_queue.cc:260] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.507494 9671 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:08.507845 9671 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:08.507937 9643 ts_tablet_manager.cc:1434] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260503 13:58:08.508176 9671 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.509029 9643 tablet_bootstrap.cc:492] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:08.513353 9728 consensus_queue.cc:1048] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:08.515988 9643 tablet_bootstrap.cc:654] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.520740 9669 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:08.523138 9644 tablet_bootstrap.cc:492] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:08.523748 9644 ts_tablet_manager.cc:1403] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.052s user 0.029s sys 0.005s
I20260503 13:58:08.526335 9644 raft_consensus.cc:359] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.527016 9644 raft_consensus.cc:385] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.523191 9669 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.527309 9644 raft_consensus.cc:740] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.531062 9644 consensus_queue.cc:260] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.535293 9669 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.535594 9644 ts_tablet_manager.cc:1434] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260503 13:58:08.536827 9671 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.538319 9643 tablet_bootstrap.cc:492] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:08.538619 9643 ts_tablet_manager.cc:1403] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.015s
I20260503 13:58:08.538611 9671 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.539321 9650 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:08.539767 9650 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.541530 9650 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.541857 9642 tablet_bootstrap.cc:492] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:08.541687 9643 raft_consensus.cc:359] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.542255 9642 ts_tablet_manager.cc:1403] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.003s
I20260503 13:58:08.544512 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:08.545090 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:08.545338 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:08.545652 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:08.545876 9601 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.542322 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:08.548333 9492 leader_election.cc:343] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.542284 9643 raft_consensus.cc:385] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.551106 9643 raft_consensus.cc:740] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.551720 9717 consensus_queue.cc:1048] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:08.551746 9643 consensus_queue.cc:260] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:08.551321 9491 leader_election.cc:343] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.552755 9491 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:08.553478 9643 ts_tablet_manager.cc:1434] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260503 13:58:08.554355 9643 tablet_bootstrap.cc:492] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:08.554894 9724 raft_consensus.cc:2749] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:08.555431 9601 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:08.555886 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:08.556244 9644 tablet_bootstrap.cc:492] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:08.558681 9642 raft_consensus.cc:359] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.559276 9642 raft_consensus.cc:385] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.559531 9642 raft_consensus.cc:740] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.560098 9643 tablet_bootstrap.cc:654] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.560719 9445 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.561108 9642 consensus_queue.cc:260] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.562980 9642 ts_tablet_manager.cc:1434] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.020s user 0.001s sys 0.005s
I20260503 13:58:08.563679 9644 tablet_bootstrap.cc:654] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.563834 9642 tablet_bootstrap.cc:492] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:08.566438 9451 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.571301 9642 tablet_bootstrap.cc:654] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.571524 9671 consensus_queue.cc:1048] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:08.574841 9643 tablet_bootstrap.cc:492] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:08.575251 9643 ts_tablet_manager.cc:1403] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.000s
I20260503 13:58:08.575559 9445 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:08.577666 9491 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:08.578398 9671 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:08.578830 9671 raft_consensus.cc:697] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:08.579566 9671 consensus_queue.cc:237] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.578043 9643 raft_consensus.cc:359] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.585032 9643 raft_consensus.cc:385] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.585299 9643 raft_consensus.cc:740] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.587424 9643 consensus_queue.cc:260] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.590564 9643 ts_tablet_manager.cc:1434] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260503 13:58:08.591815 9418 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:08.597023 9643 tablet_bootstrap.cc:492] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:08.610854 9644 tablet_bootstrap.cc:492] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:08.611344 9644 ts_tablet_manager.cc:1403] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.055s user 0.023s sys 0.006s
I20260503 13:58:08.617614 9729 raft_consensus.cc:2749] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:08.620954 9649 consensus_queue.cc:1048] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:08.621598 9601 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.622869 9644 raft_consensus.cc:359] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.624074 9644 raft_consensus.cc:385] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.624281 9644 raft_consensus.cc:740] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.624850 9643 tablet_bootstrap.cc:654] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.624859 9644 consensus_queue.cc:260] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.626899 9644 ts_tablet_manager.cc:1434] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260503 13:58:08.627732 9644 tablet_bootstrap.cc:492] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:08.631095 9649 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:08.631515 9649 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.632818 9642 tablet_bootstrap.cc:492] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:08.633183 9642 ts_tablet_manager.cc:1403] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.070s user 0.015s sys 0.022s
I20260503 13:58:08.633427 9600 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.634416 9671 consensus_queue.cc:1048] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:08.635845 9649 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:08.637231 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:08.639657 9567 leader_election.cc:343] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.640861 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:08.643567 9566 leader_election.cc:343] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.643942 9566 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:08.645035 9724 consensus_queue.cc:1048] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:08.644471 9649 raft_consensus.cc:2749] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:08.652781 9724 consensus_queue.cc:1048] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260503 13:58:08.657444 9642 raft_consensus.cc:359] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.658110 9642 raft_consensus.cc:385] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.658354 9642 raft_consensus.cc:740] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.658989 9642 consensus_queue.cc:260] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.661839 9642 ts_tablet_manager.cc:1434] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260503 13:58:08.662461 9642 tablet_bootstrap.cc:492] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:08.664659 9351 catalog_manager.cc:5671] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:08.675695 9644 tablet_bootstrap.cc:654] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.685945 9649 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:08.686472 9649 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.689170 9649 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:08.691758 9724 consensus_queue.cc:1048] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:08.692740 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:08.693392 9526 raft_consensus.cc:2468] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:08.692799 9642 tablet_bootstrap.cc:654] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.694610 9567 leader_election.cc:304] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:08.695400 9714 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:08.695822 9714 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.701021 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:08.702416 9566 leader_election.cc:343] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.703982 9714 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:08.705681 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:08.706797 9643 tablet_bootstrap.cc:492] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
W20260503 13:58:08.707023 9567 leader_election.cc:343] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:08.707576 9643 ts_tablet_manager.cc:1403] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.111s user 0.013s sys 0.016s
I20260503 13:58:08.708829 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:08.710574 9449 raft_consensus.cc:2468] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:08.711745 9566 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:08.712742 9643 raft_consensus.cc:359] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.713902 9643 raft_consensus.cc:385] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.714380 9643 raft_consensus.cc:740] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.717747 9649 raft_consensus.cc:2804] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:08.718130 9649 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:08.718433 9649 raft_consensus.cc:3060] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.715446 9643 consensus_queue.cc:260] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.720778 9669 consensus_queue.cc:1048] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:08.704514 9714 raft_consensus.cc:2804] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:08.721415 9717 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:08.721824 9717 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.723584 9643 ts_tablet_manager.cc:1434] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260503 13:58:08.723915 9714 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:08.724309 9714 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.724581 9643 tablet_bootstrap.cc:492] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:08.730108 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:08.730731 9526 raft_consensus.cc:2468] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:08.730898 9643 tablet_bootstrap.cc:654] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:08.723552 9717 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:08.732028 9567 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:08.748121 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:08.753283 9643 tablet_bootstrap.cc:492] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:08.753724 9643 ts_tablet_manager.cc:1403] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.008s
I20260503 13:58:08.756141 9643 raft_consensus.cc:359] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.757704 9643 raft_consensus.cc:385] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:08.757972 9643 raft_consensus.cc:740] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:08.758966 9643 consensus_queue.cc:260] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:08.787684 9566 leader_election.cc:343] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:08.788486 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 56553.2us
I20260503 13:58:08.788741 9832 raft_consensus.cc:2804] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:08.789667 9832 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:08.810810 9832 raft_consensus.cc:3060] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.826253 9643 ts_tablet_manager.cc:1434] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.072s user 0.004s sys 0.021s
I20260503 13:58:08.828125 9643 tablet_bootstrap.cc:492] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:08.832526 9600 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:08.850081 9714 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:08.852619 9714 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:09.277990 9832 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:09.280084 9832 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:09.993813 9837 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:09.994508 9837 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:09.996767 9836 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.014667 9836 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.016767 9836 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.017297 9836 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.017771 9836 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.026214 9836 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.027734 9837 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.095110 9837 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:10.095818 9837 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.098292 9837 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.098794 9837 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.069346 9838 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.101519 9838 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.103610 9838 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:08.963083 9650 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:10.111886 9650 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:10.126672 8304 debug-util.cc:398] Leaking SignalData structure 0x7b080000fa20 after lost signal to thread 9566
W20260503 13:58:10.127182 8304 kernel_stack_watchdog.cc:198] Thread 9566 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 376ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20260503 13:58:10.145960 9644 tablet_bootstrap.cc:492] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:10.146629 9644 ts_tablet_manager.cc:1403] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 1.519s user 0.040s sys 0.100s
I20260503 13:58:10.181670 9838 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.182328 9838 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.184806 9838 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.203651 9837 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.206291 9837 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.206861 9837 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:10.211463 9837 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:10.234243 9650 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:10.264413 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800035620 after lost signal to thread 8302
W20260503 13:58:10.336352 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800036340 after lost signal to thread 9385
I20260503 13:58:10.337083 9650 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.337612 9650 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.271402 9644 raft_consensus.cc:359] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.280682 9725 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:10.340708 9650 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:10.341013 9600 consensus_meta.cc:287] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 1.500s user 0.015s sys 0.026s
I20260503 13:58:10.341130 9725 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.341683 9650 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.342204 9650 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.342684 9600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 30) took 1536 ms. Trace:
I20260503 13:58:10.343178 9600 rpcz_store.cc:276] 0503 13:58:08.805792 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:08.807040 (+ 1248us) service_pool.cc:224] Handling call
0503 13:58:08.827314 (+ 20274us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:10.342396 (+1515082us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:10.342608 (+ 212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:10.343891 9725 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.283298 9838 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.344234 9650 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.344492 9838 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.346405 9644 raft_consensus.cc:385] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:10.346629 9644 raft_consensus.cc:740] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:10.347169 9644 consensus_queue.cc:260] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.346012 9838 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.348987 9644 ts_tablet_manager.cc:1434] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.202s user 0.006s sys 0.000s
I20260503 13:58:10.349417 9837 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:10.349833 9644 tablet_bootstrap.cc:492] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:10.355789 9644 tablet_bootstrap.cc:654] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:10.360493 9725 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:10.361280 9840 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:10.361539 9725 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.361889 9840 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.363783 9725 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.364040 9840 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.364226 9725 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.364660 9725 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.366123 9725 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.366751 9644 tablet_bootstrap.cc:492] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:10.367219 9644 ts_tablet_manager.cc:1403] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20260503 13:58:10.369702 9644 raft_consensus.cc:359] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.370795 9644 raft_consensus.cc:385] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:10.371206 9644 raft_consensus.cc:740] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:10.372022 9644 consensus_queue.cc:260] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.374197 9644 ts_tablet_manager.cc:1434] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260503 13:58:10.375221 9644 tablet_bootstrap.cc:492] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:10.378336 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 50) took 1588 ms. Trace:
I20260503 13:58:10.378705 9526 rpcz_store.cc:276] 0503 13:58:08.789959 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:10.377507 (+1587548us) service_pool.cc:224] Handling call
0503 13:58:10.377822 (+ 315us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:10.377991 (+ 169us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:10.378020 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:10.378062 (+ 42us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:10.378097 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:10.378156 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:10.378264 (+ 108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:10.380801 9644 tablet_bootstrap.cc:654] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:10.384466 9644 tablet_bootstrap.cc:492] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:10.384811 9644 ts_tablet_manager.cc:1403] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260503 13:58:10.386454 9644 raft_consensus.cc:359] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.386897 9644 raft_consensus.cc:385] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:10.387141 9644 raft_consensus.cc:740] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:10.387611 9644 consensus_queue.cc:260] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.389264 9644 ts_tablet_manager.cc:1434] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260503 13:58:10.390064 9644 tablet_bootstrap.cc:492] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:10.391215 9843 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.391762 9843 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.393996 9843 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.394615 9843 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:10.395102 9843 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.395622 9644 tablet_bootstrap.cc:654] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:10.397135 9843 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.399264 9845 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.399722 9845 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.400457 9644 tablet_bootstrap.cc:492] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:10.400900 9644 ts_tablet_manager.cc:1403] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20260503 13:58:10.401568 9845 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.402426 9643 tablet_bootstrap.cc:654] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:10.407706 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:10.408250 9523 raft_consensus.cc:3060] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:10.409886 9845 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.410315 9845 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.410508 9671 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.411055 9671 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.412038 9845 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.412478 9845 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.412896 9845 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.403430 9644 raft_consensus.cc:359] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.404585 9838 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.406637 9724 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:10.407184 9843 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:10.415271 9644 raft_consensus.cc:385] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:10.418120 9644 raft_consensus.cc:740] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:10.419121 9644 consensus_queue.cc:260] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.419410 9838 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.421222 9644 ts_tablet_manager.cc:1434] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20260503 13:58:10.421301 9838 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.422228 9644 tablet_bootstrap.cc:492] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:10.422610 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.423216 9525 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:10.418936 9845 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.427019 9567 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
W20260503 13:58:10.427564 9445 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.704s user 0.043s sys 0.082s
I20260503 13:58:10.425158 9724 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.417986 9523 raft_consensus.cc:2468] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:10.430909 9844 raft_consensus.cc:2804] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:10.441217 9844 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:10.441556 9844 raft_consensus.cc:3060] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:58:10.443468 9453 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.744s user 0.025s sys 0.087s
W20260503 13:58:10.443787 9453 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.744s user 0.025s sys 0.087s
W20260503 13:58:10.430428 9445 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.707s user 0.044s sys 0.082s
I20260503 13:58:10.445317 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 64) took 1723 ms. Trace:
I20260503 13:58:10.445254 9567 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:10.445752 9445 rpcz_store.cc:276] 0503 13:58:08.722337 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:08.722607 (+ 270us) service_pool.cc:224] Handling call
0503 13:58:08.722878 (+ 271us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:08.723168 (+ 290us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:08.723195 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:08.723587 (+ 392us) log.cc:844] Serialized 26 byte log entry
0503 13:58:10.444988 (+1721401us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:10.445030 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:10.445101 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:10.445146 (+ 45us) raft_consensus.cc:1685] finished
0503 13:58:10.445156 (+ 10us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:10.445264 (+ 108us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":70,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":83,"thread_start_us":1721068,"threads_started":1,"wal-append.queue_time_us":534390}
I20260503 13:58:10.432547 9840 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:10.447707 9840 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.448966 9836 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.449672 9840 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.434571 9837 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
W20260503 13:58:10.436266 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 1.68677e+06us
I20260503 13:58:10.450930 9837 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.453047 9837 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.439317 9724 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.440164 9671 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.460897 9844 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.455410 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.464890 9452 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:10.425241 9843 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.466907 9843 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.467382 9843 raft_consensus.cc:2804] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:10.467800 9843 raft_consensus.cc:697] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:10.468148 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:10.468748 9524 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:10.469043 9843 consensus_queue.cc:237] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.456691 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:10.473278 9450 raft_consensus.cc:3060] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:10.473840 9844 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.461637 9836 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.444474 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 61) took 1746 ms. Trace:
I20260503 13:58:10.477406 9453 rpcz_store.cc:276] 0503 13:58:08.698163 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:08.698304 (+ 141us) service_pool.cc:224] Handling call
0503 13:58:08.698637 (+ 333us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:08.698996 (+ 359us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:08.699026 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:08.699453 (+ 427us) log.cc:844] Serialized 26 byte log entry
0503 13:58:10.444119 (+1744666us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:10.444157 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:10.444237 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:10.444281 (+ 44us) raft_consensus.cc:1685] finished
0503 13:58:10.444295 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:10.444403 (+ 108us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1751,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":163,"thread_start_us":1744329,"threads_started":1,"wal-append.queue_time_us":563254}
I20260503 13:58:10.464313 9642 tablet_bootstrap.cc:492] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:10.480279 9642 ts_tablet_manager.cc:1403] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 1.818s user 0.040s sys 0.052s
I20260503 13:58:10.456487 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.481458 9644 tablet_bootstrap.cc:654] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:10.492515 9847 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.492938 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.481127 9446 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
W20260503 13:58:10.495033 9565 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.746s user 0.011s sys 0.029s
I20260503 13:58:10.490326 9849 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.490490 9671 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.491998 9729 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
W20260503 13:58:10.501403 9565 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.754s user 0.011s sys 0.030s
I20260503 13:58:10.493038 9847 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.508883 9843 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:10.509395 9843 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.510443 9847 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.511514 9843 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.516590 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.501262 9671 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.522771 9671 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.525436 9847 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:10.503675 9729 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.504551 9849 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.700682 9858 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:10.601293 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.701385 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.701723 9847 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.697798 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.599205 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.606266 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.695577 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.703678 9847 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.590790 9845 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.694204 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.704502 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.693485 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.597062 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.581133 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:10.634680 9382 debug-util.cc:398] Leaking SignalData structure 0x7b080025f1c0 after lost signal to thread 9477
I20260503 13:58:10.705657 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.706179 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.592626 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:10.591979 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:10.707099 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:10.526003 9649 consensus_meta.cc:287] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 1.807s user 0.017s sys 0.032s
I20260503 13:58:10.584115 9350 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51410 (request call id 44) took 1892 ms. Trace:
I20260503 13:58:10.590950 9650 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:10.632165 9838 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:10.704757 9602 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:10.689967 9859 consensus_queue.cc:1048] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260503 13:58:10.593084 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.704469 9845 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.715384 9452 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:10.701373 9858 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:10.608021 9642 raft_consensus.cc:359] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.716274 9642 raft_consensus.cc:385] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:10.716650 9642 raft_consensus.cc:740] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:10.708384 9350 rpcz_store.cc:276] 0503 13:58:08.691846 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:08.692012 (+ 166us) service_pool.cc:224] Handling call
0503 13:58:10.584031 (+1892019us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":2}
I20260503 13:58:10.718407 9642 consensus_queue.cc:260] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.706717 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.711866 9643 tablet_bootstrap.cc:492] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:10.722939 9855 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.594311 9524 raft_consensus.cc:2468] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:10.723489 9855 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.712826 9849 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.713534 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.723146 9643 ts_tablet_manager.cc:1403] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 1.896s user 0.043s sys 0.045s
I20260503 13:58:10.714044 9446 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:10.714389 9843 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:10.724936 9843 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.724105 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.726522 9524 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:10.721669 9644 tablet_bootstrap.cc:492] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:10.713878 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:10.706717 9382 debug-util.cc:398] Leaking SignalData structure 0x7b080025fa00 after lost signal to thread 9552
I20260503 13:58:10.709295 9649 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.525348 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:10.727212 9644 ts_tablet_manager.cc:1403] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.305s user 0.014s sys 0.008s
I20260503 13:58:10.636699 9839 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:10.640617 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:10.731626 9855 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.665680 9850 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.655373 9844 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:10.733160 9597 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:10.736771 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.707237 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.579468 9852 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.565621 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.628731 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.672365 9836 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.526300 9450 raft_consensus.cc:2468] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:10.712174 9838 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.530953 9526 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:10.715939 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.734473 9523 raft_consensus.cc:3060] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:10.729421 9642 ts_tablet_manager.cc:1434] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.249s user 0.016s sys 0.005s
I20260503 13:58:10.739415 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.740907 9844 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.742499 9729 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.726764 9843 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.713204 9650 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.762704 9644 raft_consensus.cc:359] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.763350 9644 raft_consensus.cc:385] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:10.763621 9650 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.633435 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.746624 9643 raft_consensus.cc:359] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.569222 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.720791 9847 consensus_queue.cc:1048] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260503 13:58:10.739892 9839 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.750025 9642 tablet_bootstrap.cc:492] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:10.745440 9649 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.752887 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.751987 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.753499 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:10.746541 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264ec0 after lost signal to thread 9626
I20260503 13:58:10.744717 9855 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:10.756300 9856 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.752893 9838 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.756467 9843 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:10.760121 9844 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.761713 9858 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:10.763625 9644 raft_consensus.cc:740] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:10.764846 9643 raft_consensus.cc:385] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:10.755057 9852 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.747372 9528 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:10.769845 9644 consensus_queue.cc:260] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.770109 9643 raft_consensus.cc:740] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
W20260503 13:58:10.768458 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264ea0 after lost signal to thread 9627
I20260503 13:58:10.769500 9843 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.771970 9856 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.767112 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.765584 9523 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:10.784909 9643 consensus_queue.cc:260] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.787397 9856 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.789319 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.789738 9843 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.769778 9446 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:10.796586 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.796722 9598 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:10.798264 9451 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:10.799187 9449 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:10.800769 9839 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.802675 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.803067 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.803880 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.804211 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.805409 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.805531 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.806725 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:10.806993 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.806759 9642 tablet_bootstrap.cc:654] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:10.808522 9567 leader_election.cc:304] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:10.770399 9855 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.770697 9597 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:10.770659 9845 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.814183 9864 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.815855 9644 ts_tablet_manager.cc:1434] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.083s user 0.003s sys 0.007s
I20260503 13:58:10.816534 9644 tablet_bootstrap.cc:492] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:10.770859 9526 raft_consensus.cc:2468] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:10.810639 9855 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.822054 9843 raft_consensus.cc:2804] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
W20260503 13:58:10.812968 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264bc0 after lost signal to thread 9732
W20260503 13:58:10.825008 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264b80 after lost signal to thread 9740
W20260503 13:58:10.825289 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264b60 after lost signal to thread 9741
W20260503 13:58:10.825518 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264b40 after lost signal to thread 9742
I20260503 13:58:10.814694 9864 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.824242 9843 raft_consensus.cc:697] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:10.826402 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.827162 9843 consensus_queue.cc:237] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.828773 9601 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
W20260503 13:58:10.825765 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264ac0 after lost signal to thread 9768
I20260503 13:58:10.745889 9850 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.834163 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.834283 9602 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:10.835374 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.835068 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.837108 9601 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:10.837433 9596 raft_consensus.cc:2393] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:10.833546 9567 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:10.837204 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.838203 9855 raft_consensus.cc:2804] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:10.838346 9597 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:10.838613 9855 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:10.813776 9418 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:10.838944 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.839164 9855 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:10.839413 9845 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:10.839541 9598 raft_consensus.cc:2468] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:10.839792 9845 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.840189 9845 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:10.840406 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:10.833150 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264aa0 after lost signal to thread 9770
I20260503 13:58:10.845564 9643 ts_tablet_manager.cc:1434] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.121s user 0.011s sys 0.007s
I20260503 13:58:10.846540 9643 tablet_bootstrap.cc:492] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:10.850368 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.852903 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.853161 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.853587 9845 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.854525 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.858496 9845 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.859268 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.858911 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.860101 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.860682 9449 raft_consensus.cc:2393] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:10.861058 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.861217 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.862524 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.862668 9451 raft_consensus.cc:2468] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
W20260503 13:58:10.844746 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264a40 after lost signal to thread 9774
I20260503 13:58:10.858845 9567 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:10.868362 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:10.867529 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264980 after lost signal to thread 9781
I20260503 13:58:10.868944 9451 raft_consensus.cc:3060] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:10.872040 9865 raft_consensus.cc:2804] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:10.872524 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.872886 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:10.873281 9852 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.872589 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.876073 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.876619 9865 raft_consensus.cc:697] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:10.877532 9598 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
W20260503 13:58:10.874501 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264800 after lost signal to thread 9793
I20260503 13:58:10.878364 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.878033 9865 consensus_queue.cc:237] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:10.882033 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264820 after lost signal to thread 9800
I20260503 13:58:10.895737 9671 consensus_queue.cc:1048] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260503 13:58:10.896090 9492 leader_election.cc:343] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:10.877272 9449 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:10.896919 9492 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:10.899230 9567 leader_election.cc:343] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:10.880972 9850 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.900288 9448 raft_consensus.cc:1240] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:10.905488 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.900646 9724 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:10.899230 9864 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:10.906132 9724 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.906517 9724 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:10.909638 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:10.910851 9418 leader_election.cc:343] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:10.911823 9642 tablet_bootstrap.cc:492] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:10.912165 9642 ts_tablet_manager.cc:1403] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.163s user 0.012s sys 0.009s
W20260503 13:58:10.898698 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264760 after lost signal to thread 9803
I20260503 13:58:10.911273 9492 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:10.915041 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:10.917488 9567 leader_election.cc:343] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:10.918490 9566 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:10.918895 9451 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:10.923344 9642 raft_consensus.cc:359] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.924041 9642 raft_consensus.cc:385] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:10.924302 9642 raft_consensus.cc:740] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:10.919561 9855 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.920413 9418 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:10.922312 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264740 after lost signal to thread 9807
W20260503 13:58:10.926384 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264720 after lost signal to thread 9808
W20260503 13:58:10.926708 9382 debug-util.cc:398] Leaking SignalData structure 0x7b08002646e0 after lost signal to thread 9810
W20260503 13:58:10.927075 9382 debug-util.cc:398] Leaking SignalData structure 0x7b08002646c0 after lost signal to thread 9811
I20260503 13:58:10.927466 9492 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:10.926771 9840 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:10.928139 9840 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.928097 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.928356 9849 raft_consensus.cc:2764] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260503 13:58:10.928632 9840 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:10.924935 9642 consensus_queue.cc:260] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.934346 9642 ts_tablet_manager.cc:1434] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260503 13:58:10.933876 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.927718 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.936906 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.937579 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:10.938179 9446 raft_consensus.cc:3060] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:10.937129 9452 raft_consensus.cc:2393] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
W20260503 13:58:10.933292 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264660 after lost signal to thread 9815
I20260503 13:58:10.940819 9418 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:10.938311 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.945456 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.943075 9449 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:10.947611 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.941107 9492 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:10.935931 9845 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:10.952807 9845 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.953020 9671 raft_consensus.cc:2804] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:10.940815 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.935075 9843 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:10.944104 9642 tablet_bootstrap.cc:492] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:10.944779 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:10.948660 9847 raft_consensus.cc:2804] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:10.953509 9837 consensus_queue.cc:1048] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:10.953656 9845 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:10.954658 9847 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.955070 9847 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
W20260503 13:58:10.949257 9418 leader_election.cc:343] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:10.957290 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.957803 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:10.943559 9382 debug-util.cc:398] Leaking SignalData structure 0x7b08002645e0 after lost signal to thread 9816
W20260503 13:58:10.961537 9492 leader_election.cc:343] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:10.953995 9843 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:10.957871 9449 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:10.965293 9724 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:10.966048 9382 debug-util.cc:398] Leaking SignalData structure 0x7b0800264580 after lost signal to thread 9825
I20260503 13:58:10.966665 9847 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.968106 9845 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.968626 9843 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:10.970295 9845 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.949302 9840 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:10.961433 9418 leader_election.cc:343] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:10.978094 9855 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:10.978534 9418 leader_election.cc:343] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:10.979379 9642 tablet_bootstrap.cc:654] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:10.951756 9446 raft_consensus.cc:2468] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:10.972921 9847 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.973834 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:10.980911 9644 tablet_bootstrap.cc:654] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:10.953588 9671 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.981722 9671 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:10.981938 9492 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:10.982748 9837 raft_consensus.cc:2804] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:10.983140 9837 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:10.983366 9643 tablet_bootstrap.cc:654] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:10.983574 9837 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:10.975276 9843 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:10.984944 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.987594 9840 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:10.998077 9852 consensus_queue.cc:1059] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260503 13:58:11.010804 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.012938 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.997049 9724 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.015540 9865 consensus_queue.cc:1048] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:11.017779 9867 raft_consensus.cc:3055] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.018153 9867 raft_consensus.cc:740] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:11.018957 9867 consensus_queue.cc:260] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.020242 9867 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.021595 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:11.024180 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:11.025410 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.031152 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:10.995047 9837 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.017324 9843 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:11.039722 9840 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:11.040369 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.040673 9840 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.040972 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:11.041272 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:11.041855 9671 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.043001 9840 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.043641 9671 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.035001 9566 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.045703 9566 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.048400 9642 tablet_bootstrap.cc:492] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:11.048949 9642 ts_tablet_manager.cc:1403] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.105s user 0.009s sys 0.037s
I20260503 13:58:11.049284 9837 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.049348 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.051265 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:11.052278 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:11.058527 9450 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:11.048868 9843 raft_consensus.cc:2749] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.056933 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:11.057180 9855 raft_consensus.cc:2749] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.061170 9492 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:11.062464 9599 raft_consensus.cc:3060] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:11.062981 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:11.063529 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:11.063865 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:11.064070 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:11.064680 9598 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:11.067552 9724 raft_consensus.cc:2804] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:11.068107 9724 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:11.068783 9724 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:11.065415 9566 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.071278 9566 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.071978 9843 raft_consensus.cc:2749] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.072376 9566 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.072579 9843 raft_consensus.cc:2749] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.073697 9566 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.074415 9843 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:11.074738 9843 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:11.075181 9843 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.078243 9724 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.080672 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:11.081210 9451 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:11.082432 9843 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.082912 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.089540 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:11.088636 9642 raft_consensus.cc:359] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.090462 9642 raft_consensus.cc:385] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:11.090731 9642 raft_consensus.cc:740] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:11.091837 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:11.074738 9566 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.092839 9451 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:11.093084 9566 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.093303 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.094152 9566 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.094604 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.094772 9865 raft_consensus.cc:2804] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:11.095144 9865 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:11.095441 9865 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:11.089807 9843 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:11.097216 9867 raft_consensus.cc:2749] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.097652 9852 raft_consensus.cc:2804] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:11.100332 9852 raft_consensus.cc:697] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:11.101219 9852 consensus_queue.cc:237] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.101635 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:11.098276 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.099247 9642 consensus_queue.cc:260] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.100241 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:11.108975 9492 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:11.109968 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:11.110061 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.111102 9644 tablet_bootstrap.cc:492] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:11.111519 9644 ts_tablet_manager.cc:1403] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.295s user 0.004s sys 0.012s
W20260503 13:58:11.110409 9416 leader_election.cc:343] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.113042 9416 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.114439 9847 raft_consensus.cc:2749] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:11.114715 9416 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.115419 9847 raft_consensus.cc:2749] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.116336 9452 raft_consensus.cc:1240] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:11.116647 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:11.117648 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:11.116276 9416 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.119176 9845 raft_consensus.cc:2749] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.119205 9566 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.119886 9671 raft_consensus.cc:2804] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:11.120240 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:11.120352 9566 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.120609 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:11.120354 9671 raft_consensus.cc:697] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:11.121726 9671 consensus_queue.cc:237] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.122476 9644 raft_consensus.cc:359] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.123126 9644 raft_consensus.cc:385] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:11.123395 9644 raft_consensus.cc:740] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:11.123975 9644 consensus_queue.cc:260] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.125495 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.126042 9855 raft_consensus.cc:2749] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:11.126204 9864 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:11.129861 9416 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.125385 9350 catalog_manager.cc:5671] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:11.129733 9864 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.133064 9350 catalog_manager.cc:5671] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:11.134828 9350 catalog_manager.cc:5671] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:11.137614 9642 ts_tablet_manager.cc:1434] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.088s user 0.004s sys 0.003s
I20260503 13:58:11.140439 9643 tablet_bootstrap.cc:492] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:11.140537 9642 tablet_bootstrap.cc:492] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:11.140913 9643 ts_tablet_manager.cc:1403] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.295s user 0.013s sys 0.013s
I20260503 13:58:11.132050 9867 raft_consensus.cc:2749] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.139678 9847 raft_consensus.cc:2804] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:11.142064 9644 ts_tablet_manager.cc:1434] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20260503 13:58:11.132464 9843 raft_consensus.cc:2749] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.143579 9864 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:11.145844 9644 tablet_bootstrap.cc:492] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:11.133041 9724 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.144611 9847 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:11.153194 9526 raft_consensus.cc:1240] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:11.148613 9867 consensus_queue.cc:1048] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260503 13:58:11.153832 9864 consensus_queue.cc:1059] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:11.153223 9566 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.147761 9840 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:11.155938 9849 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:11.158298 9850 raft_consensus.cc:3055] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.158432 9416 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.155951 9840 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.159356 9845 raft_consensus.cc:2804] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:11.159704 9845 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:11.160044 9845 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.160192 9850 raft_consensus.cc:740] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:11.156370 9849 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.161675 9566 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.164317 9566 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.165062 9416 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:11.165378 9491 consensus_peers.cc:597] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a -> Peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Couldn't send request to peer 4bea375a55034750a1f8804b0db97868. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260503 13:58:11.154744 9847 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.166625 9859 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:11.166687 9867 consensus_queue.cc:1059] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:11.167050 9859 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:11.167419 9859 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.167449 9855 raft_consensus.cc:2749] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.167651 9867 raft_consensus.cc:3055] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.168033 9867 raft_consensus.cc:740] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:11.170259 9643 raft_consensus.cc:359] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.170931 9643 raft_consensus.cc:385] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:11.171190 9643 raft_consensus.cc:740] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:11.171757 9643 consensus_queue.cc:260] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.154445 9865 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.176002 9865 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:11.176168 9850 consensus_queue.cc:260] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.177125 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.177006 9845 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.177449 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:11.177789 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:11.162091 9864 raft_consensus.cc:2749] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260503 13:58:11.178371 9567 leader_election.cc:343] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.177480 9850 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.161341 9840 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.179463 9867 consensus_queue.cc:260] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:11.179775 9567 leader_election.cc:343] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.209570 9643 ts_tablet_manager.cc:1434] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.068s user 0.009s sys 0.000s
W20260503 13:58:11.209707 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 45468.1us
I20260503 13:58:11.209193 9864 raft_consensus.cc:2749] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.212975 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.213712 9845 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.214524 9416 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.218338 9643 tablet_bootstrap.cc:492] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:11.217156 9491 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.220736 9845 raft_consensus.cc:2804] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:11.219066 9566 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.221107 9845 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:11.221870 9845 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.221899 9864 raft_consensus.cc:2749] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.224038 9840 raft_consensus.cc:2749] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260503 13:58:11.209930 9492 leader_election.cc:343] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.228223 9642 tablet_bootstrap.cc:654] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:11.230501 9867 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.234086 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:11.234755 9449 raft_consensus.cc:2468] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:11.236817 9845 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.238978 9566 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.239581 9445 raft_consensus.cc:1240] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:11.239902 9865 raft_consensus.cc:2804] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:11.240255 9865 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:11.240552 9865 raft_consensus.cc:3060] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:11.242686 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:11.243300 9451 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:11.240589 9491 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.245410 9840 raft_consensus.cc:2749] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.246275 9526 raft_consensus.cc:1240] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
W20260503 13:58:11.246351 9491 leader_election.cc:343] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.246796 9491 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.247541 9840 raft_consensus.cc:2749] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:11.247690 9852 consensus_queue.cc:1059] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:11.248584 9852 raft_consensus.cc:3055] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.248930 9852 raft_consensus.cc:740] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:11.249336 9845 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.228191 9849 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.249637 9852 consensus_queue.cc:260] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.252194 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:11.229878 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 47591.1us
I20260503 13:58:11.248350 9865 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.260254 9525 raft_consensus.cc:1240] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:11.262435 9526 raft_consensus.cc:1240] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:11.262657 9852 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.262812 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:11.260504 9416 leader_election.cc:343] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.266314 9864 consensus_queue.cc:1059] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260503 13:58:11.263929 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:11.268949 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.275630 9864 consensus_queue.cc:1059] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:11.276278 9865 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:11.276527 9864 raft_consensus.cc:3055] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.276721 9855 raft_consensus.cc:3055] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.276932 9864 raft_consensus.cc:740] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:11.277055 9855 raft_consensus.cc:740] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:11.279487 9864 consensus_queue.cc:260] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.279810 9855 consensus_queue.cc:260] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.280308 9451 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:11.281016 9864 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
W20260503 13:58:11.282145 9880 log.cc:927] Time spent T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260503 13:58:11.281090 9855 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.282824 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:11.284070 9567 leader_election.cc:343] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.285236 9847 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.286628 9445 raft_consensus.cc:1240] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:11.287317 9847 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:11.277352 9492 leader_election.cc:343] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.290354 9840 consensus_queue.cc:1059] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:11.291261 9849 raft_consensus.cc:3055] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.291615 9849 raft_consensus.cc:740] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:11.292999 9849 consensus_queue.cc:260] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.294142 9849 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.299158 9859 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.301064 9859 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.304121 9643 tablet_bootstrap.cc:654] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:11.305333 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:11.306412 9450 raft_consensus.cc:3060] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:11.270937 9416 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:11.309340 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 51055.9us
I20260503 13:58:11.310047 9416 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:11.311859 9416 leader_election.cc:343] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.313115 9644 tablet_bootstrap.cc:654] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:11.313702 9839 raft_consensus.cc:2749] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.315083 9566 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.316478 9864 raft_consensus.cc:2804] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:11.312235 9416 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.321081 9864 raft_consensus.cc:697] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:11.322974 9864 consensus_queue.cc:237] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.333539 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.333945 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.334465 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:11.336865 9859 raft_consensus.cc:2749] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:11.337009 9491 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.339042 9837 raft_consensus.cc:2749] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260503 13:58:11.339630 9491 leader_election.cc:343] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.339860 9837 consensus_queue.cc:1048] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:11.340061 9491 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.340752 9725 consensus_queue.cc:1048] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:11.342368 9416 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.343418 9416 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.318169 9847 raft_consensus.cc:2804] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:11.344805 9847 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:11.345153 9847 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.347759 9491 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.348739 9840 raft_consensus.cc:2749] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:11.349370 9839 raft_consensus.cc:2804] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:11.349712 9839 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:11.350061 9839 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.348767 9671 raft_consensus.cc:2749] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.344484 9416 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.350464 9491 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:11.338662 9598 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:11.356745 9671 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:11.357105 9671 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:11.357455 9671 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.353989 9859 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:11.361042 9859 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:11.361425 9859 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.356760 9845 raft_consensus.cc:2749] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:11.364735 9856 consensus_queue.cc:1048] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:11.365073 9450 raft_consensus.cc:2468] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:11.367357 9450 raft_consensus.cc:1240] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:11.368692 9566 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.372159 9599 raft_consensus.cc:2468] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:11.368690 9837 consensus_queue.cc:1059] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260503 13:58:11.372058 9859 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.383565 9416 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.382213 9858 raft_consensus.cc:2804] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:11.384653 9845 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:11.384380 9491 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.385041 9845 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:11.385390 9840 raft_consensus.cc:2749] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.385771 9845 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.386598 9491 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.385028 9858 raft_consensus.cc:697] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:11.387543 9840 raft_consensus.cc:2749] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.387861 9858 consensus_queue.cc:237] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.360349 9521 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:11.384984 9416 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.393491 9838 raft_consensus.cc:2804] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:11.393883 9838 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:11.394219 9838 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.397440 9839 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.398406 9450 raft_consensus.cc:1240] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:11.395776 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.400542 9445 raft_consensus.cc:1240] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:11.388841 9491 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.401530 9845 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.401772 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.402403 9849 raft_consensus.cc:3055] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.402765 9849 raft_consensus.cc:740] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:11.385790 9859 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.406854 9644 tablet_bootstrap.cc:492] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:11.407200 9644 ts_tablet_manager.cc:1403] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.262s user 0.007s sys 0.006s
I20260503 13:58:11.404829 9837 raft_consensus.cc:2749] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.408648 9849 consensus_queue.cc:260] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.409624 9837 consensus_queue.cc:1059] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:11.409701 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.410574 9837 raft_consensus.cc:3055] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.410420 9847 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.409825 9849 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.410035 9491 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.410925 9837 raft_consensus.cc:740] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:11.411502 9839 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.411830 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.401072 9416 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.415807 9840 raft_consensus.cc:2749] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.426258 9725 consensus_queue.cc:1059] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:11.433319 9724 raft_consensus.cc:3055] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.433706 9724 raft_consensus.cc:740] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:11.433791 9839 raft_consensus.cc:2749] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.434242 9416 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.427934 9445 raft_consensus.cc:1240] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:11.423043 9418 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:11.417603 9491 leader_election.cc:343] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.412060 9837 consensus_queue.cc:260] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.422657 9644 raft_consensus.cc:359] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.436419 9644 raft_consensus.cc:385] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:11.424937 9838 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.437546 9837 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.436705 9644 raft_consensus.cc:740] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:11.434535 9724 consensus_queue.cc:260] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.440636 9724 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.444728 9845 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.439026 9644 consensus_queue.cc:260] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.439949 9839 raft_consensus.cc:2804] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:11.445482 9847 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.448491 9864 consensus_queue.cc:1048] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260503 13:58:11.452824 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.446429 9845 raft_consensus.cc:2749] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.446247 9416 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.446715 9840 consensus_queue.cc:1059] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260503 13:58:11.436721 9671 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.472795 9839 raft_consensus.cc:697] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:11.477818 9839 consensus_queue.cc:237] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.478937 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:11.480015 9445 raft_consensus.cc:2393] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:11.481151 9642 tablet_bootstrap.cc:492] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:11.481644 9642 ts_tablet_manager.cc:1403] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.341s user 0.009s sys 0.017s
I20260503 13:58:11.485167 9643 tablet_bootstrap.cc:492] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:11.485510 9643 ts_tablet_manager.cc:1403] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.267s user 0.012s sys 0.009s
I20260503 13:58:11.460434 9491 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.450525 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.449357 9838 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.487895 9644 ts_tablet_manager.cc:1434] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.081s user 0.006s sys 0.002s
I20260503 13:58:11.482122 9845 raft_consensus.cc:2764] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260503 13:58:11.497254 9644 tablet_bootstrap.cc:492] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:11.494180 9725 raft_consensus.cc:2749] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.498229 9724 consensus_queue.cc:1048] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260503 13:58:11.505084 9856 consensus_queue.cc:1048] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:11.505458 9642 raft_consensus.cc:359] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.493645 9416 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.506166 9642 raft_consensus.cc:385] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:11.506456 9642 raft_consensus.cc:740] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:11.507014 9642 consensus_queue.cc:260] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.501964 9671 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.510726 9839 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:11.511162 9839 raft_consensus.cc:697] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:11.511888 9839 consensus_queue.cc:237] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.512362 9445 raft_consensus.cc:1240] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:11.513353 9837 raft_consensus.cc:3055] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.513686 9837 raft_consensus.cc:740] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:11.520360 9840 consensus_queue.cc:1059] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:11.521299 9840 raft_consensus.cc:3055] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.521618 9840 raft_consensus.cc:740] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:11.520892 9837 consensus_queue.cc:260] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.522784 9642 ts_tablet_manager.cc:1434] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.041s user 0.006s sys 0.001s
I20260503 13:58:11.542091 9642 tablet_bootstrap.cc:492] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:11.509618 9858 consensus_queue.cc:1048] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260503 13:58:11.524916 9859 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:11.543910 9416 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.566721 9451 raft_consensus.cc:1240] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
W20260503 13:58:11.566653 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 47024us
I20260503 13:58:11.568775 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.569010 9845 raft_consensus.cc:2804] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:11.569540 9845 raft_consensus.cc:697] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:11.570350 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.570385 9845 consensus_queue.cc:237] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.563673 9859 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.577548 9859 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.579226 9671 consensus_queue.cc:1059] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:11.580119 9724 raft_consensus.cc:3055] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.580957 9724 raft_consensus.cc:740] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:11.581681 9724 consensus_queue.cc:260] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.555323 9840 consensus_queue.cc:260] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.634541 9643 raft_consensus.cc:359] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.637665 9643 raft_consensus.cc:385] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:11.638170 9643 raft_consensus.cc:740] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:11.638815 9643 consensus_queue.cc:260] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:11.544085 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 55069.9us
I20260503 13:58:11.636888 9840 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.633836 9724 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.642588 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.645781 9451 raft_consensus.cc:1240] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:11.648792 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.648909 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.650132 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:11.650521 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.651209 9597 raft_consensus.cc:2393] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 2: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:11.649883 9642 tablet_bootstrap.cc:654] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:11.653074 9643 ts_tablet_manager.cc:1434] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.167s user 0.010s sys 0.002s
I20260503 13:58:11.654098 9643 tablet_bootstrap.cc:492] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:11.666126 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.669657 9644 tablet_bootstrap.cc:654] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:11.674360 9416 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.674942 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.677384 9849 consensus_queue.cc:1059] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260503 13:58:11.682461 9896 log.cc:927] Time spent T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
I20260503 13:58:11.646612 9491 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.681396 9839 raft_consensus.cc:2749] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.684543 9849 raft_consensus.cc:2749] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.683683 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.685132 9598 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.680778 9724 raft_consensus.cc:3055] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.687276 9724 raft_consensus.cc:740] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:11.693079 9416 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.693378 9837 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.693832 9859 raft_consensus.cc:2749] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.697172 9643 tablet_bootstrap.cc:654] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:11.697878 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:11.698423 9724 consensus_queue.cc:260] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.700305 9898 consensus_queue.cc:1048] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
W20260503 13:58:11.692724 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 49039.3us
W20260503 13:58:11.707161 9491 leader_election.cc:343] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.699630 9724 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.709836 9864 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:11.710263 9864 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.712028 9864 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:11.713814 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:11.721400 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:11.724159 9567 consensus_peers.cc:597] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 -> Peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Couldn't send request to peer 428b342dc4f041df956015e7d7cd173a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260503 13:58:11.725696 9567 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:11.726315 9491 leader_election.cc:343] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.732826 9416 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.733253 9491 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.733641 9599 raft_consensus.cc:1240] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:11.734071 9859 raft_consensus.cc:2749] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260503 13:58:11.734431 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:11.727713 9864 raft_consensus.cc:2749] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:11.736191 9849 raft_consensus.cc:2749] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:11.737879 9416 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.738888 9859 raft_consensus.cc:2749] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.741092 9451 raft_consensus.cc:1240] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:11.741130 9416 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.741943 9859 raft_consensus.cc:2749] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260503 13:58:11.720772 9418 leader_election.cc:343] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.744247 9859 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:11.744678 9859 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.747494 9859 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.748947 9864 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:11.749985 9864 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.727200 9598 raft_consensus.cc:2468] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:11.753135 9864 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:11.754333 9416 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.755121 9839 raft_consensus.cc:2749] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.754379 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:11.755555 9416 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.756346 9839 raft_consensus.cc:2749] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:11.760517 9642 tablet_bootstrap.cc:492] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:11.761022 9642 ts_tablet_manager.cc:1403] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.219s user 0.017s sys 0.010s
I20260503 13:58:11.759318 9858 consensus_queue.cc:1048] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260503 13:58:11.755944 9445 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
W20260503 13:58:11.771155 9416 leader_election.cc:343] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.772015 9416 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.775915 9566 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:11.759846 9837 consensus_queue.cc:1059] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260503 13:58:11.778463 9858 raft_consensus.cc:2804] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:11.779788 9858 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:11.780529 9858 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:11.775053 9350 catalog_manager.cc:5671] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:11.782987 9350 catalog_manager.cc:5671] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:11.782924 9839 raft_consensus.cc:2749] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:11.783417 9855 consensus_queue.cc:1048] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260503 13:58:11.785825 9350 catalog_manager.cc:5671] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:11.787390 9642 raft_consensus.cc:359] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.787880 9642 raft_consensus.cc:385] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:11.788072 9642 raft_consensus.cc:740] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:11.788987 9642 consensus_queue.cc:260] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.782889 9724 raft_consensus.cc:3055] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.790872 9724 raft_consensus.cc:740] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:11.791746 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:11.791867 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:11.792371 9526 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:11.792205 9724 consensus_queue.cc:260] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.793210 9724 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.793793 9839 consensus_queue.cc:1059] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:11.794585 9845 raft_consensus.cc:3055] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:11.794900 9845 raft_consensus.cc:740] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:11.790927 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:11.795862 9521 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:11.795629 9845 consensus_queue.cc:260] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.796811 9845 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.794948 9416 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.797850 9839 raft_consensus.cc:2804] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election won for term 2
W20260503 13:58:11.798308 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:11.798877 9642 ts_tablet_manager.cc:1434] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.038s user 0.005s sys 0.001s
I20260503 13:58:11.798270 9839 raft_consensus.cc:697] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:11.800067 9644 tablet_bootstrap.cc:492] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:11.801599 9644 ts_tablet_manager.cc:1403] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.305s user 0.013s sys 0.008s
I20260503 13:58:11.800836 9839 consensus_queue.cc:237] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.808315 9855 consensus_queue.cc:1048] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:11.807667 9858 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.812391 9642 tablet_bootstrap.cc:492] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:11.813829 9418 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:11.814611 9598 raft_consensus.cc:2393] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:11.814635 9839 raft_consensus.cc:2804] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:11.815567 9839 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:11.815902 9839 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:11.816141 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:11.814895 9858 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:11.818133 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:11.818867 9521 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:11.822644 9839 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.823800 9449 raft_consensus.cc:2393] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:11.825101 9839 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.825318 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:11.826067 9494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:11.827527 9643 tablet_bootstrap.cc:492] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:11.828020 9643 ts_tablet_manager.cc:1403] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.174s user 0.009s sys 0.010s
I20260503 13:58:11.828115 9349 catalog_manager.cc:5671] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 1 to 2, leader changed from 428b342dc4f041df956015e7d7cd173a (127.8.26.194) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 2 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:11.830204 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:11.830687 9349 catalog_manager.cc:5671] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 1 to 2, leader changed from 4bea375a55034750a1f8804b0db97868 (127.8.26.195) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 2 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:11.832587 9349 catalog_manager.cc:5671] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } }
I20260503 13:58:11.834396 9349 catalog_manager.cc:5671] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 1 to 2, leader changed from 4bea375a55034750a1f8804b0db97868 (127.8.26.195) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 2 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:11.839375 9521 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:11.840476 9567 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:11.841343 9858 raft_consensus.cc:2804] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:11.841846 9858 raft_consensus.cc:697] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:11.842622 9858 consensus_queue.cc:237] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.845346 9643 raft_consensus.cc:359] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.845954 9643 raft_consensus.cc:385] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:11.846215 9643 raft_consensus.cc:740] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:11.846787 9643 consensus_queue.cc:260] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.848827 9643 ts_tablet_manager.cc:1434] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.019s user 0.003s sys 0.003s
I20260503 13:58:11.849704 9643 tablet_bootstrap.cc:492] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:11.850533 9599 raft_consensus.cc:1240] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260503 13:58:11.842595 9644 raft_consensus.cc:359] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:11.852777 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:11.852690 9644 raft_consensus.cc:385] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260503 13:58:11.849081 9494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:11.864104 9644 raft_consensus.cc:740] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:11.865549 9644 consensus_queue.cc:260] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.853585 9521 raft_consensus.cc:3060] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:11.868765 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.869887 9644 ts_tablet_manager.cc:1434] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.068s user 0.006s sys 0.010s
W20260503 13:58:11.874641 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:11.879683 9598 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:58:11.881233 9491 leader_election.cc:343] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.882982 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:11.853466 9599 raft_consensus.cc:1240] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260503 13:58:11.891297 9644 tablet_bootstrap.cc:492] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:11.902622 9349 catalog_manager.cc:5671] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:11.898728 9491 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:11.906366 9494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:11.911423 9724 raft_consensus.cc:2749] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:11.907070 9416 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:11.914997 9839 raft_consensus.cc:2749] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260503 13:58:11.917352 9526 raft_consensus.cc:1275] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:11.921445 9644 tablet_bootstrap.cc:654] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:11.921312 9597 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:11.918633 9839 consensus_queue.cc:1048] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260503 13:58:11.928973 9491 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:11.929813 9724 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:11.930253 9724 raft_consensus.cc:697] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:11.923964 9351 catalog_manager.cc:5671] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 1 to 2, leader changed from 428b342dc4f041df956015e7d7cd173a (127.8.26.194) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 2 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:11.930994 9724 consensus_queue.cc:237] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.931349 9850 consensus_queue.cc:1048] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:11.931872 9597 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:11.939370 9643 tablet_bootstrap.cc:654] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:11.939816 9525 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:11.941254 9839 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:11.941711 9839 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.946190 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:11.946718 9642 tablet_bootstrap.cc:654] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
W20260503 13:58:11.947721 9418 leader_election.cc:343] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:11.950311 9837 consensus_queue.cc:1048] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:11.971894 9644 tablet_bootstrap.cc:492] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:11.972318 9644 ts_tablet_manager.cc:1403] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.081s user 0.011s sys 0.003s
I20260503 13:58:11.974926 9644 raft_consensus.cc:359] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.975531 9644 raft_consensus.cc:385] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:11.978478 9526 raft_consensus.cc:1275] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:11.981326 9839 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:11.983441 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:11.984093 9601 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:11.984054 9838 consensus_queue.cc:1048] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:11.985769 9416 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:11.986816 9644 raft_consensus.cc:740] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:11.987471 9644 consensus_queue.cc:260] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:11.990975 9858 consensus_queue.cc:1048] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:11.988740 9847 raft_consensus.cc:2804] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:11.993301 9847 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:11.993626 9847 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:11.998198 9519 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:12.004714 9644 ts_tablet_manager.cc:1434] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.032s user 0.006s sys 0.001s
I20260503 13:58:12.005592 9644 tablet_bootstrap.cc:492] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:12.011703 9642 tablet_bootstrap.cc:492] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:12.012106 9642 ts_tablet_manager.cc:1403] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.200s user 0.006s sys 0.010s
I20260503 13:58:12.015373 9519 raft_consensus.cc:1275] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:12.016777 9838 consensus_queue.cc:1048] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:12.019296 9642 raft_consensus.cc:359] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.019917 9642 raft_consensus.cc:385] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.020169 9642 raft_consensus.cc:740] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.026576 9597 raft_consensus.cc:1275] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:12.028723 9838 consensus_queue.cc:1048] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:12.044409 9642 consensus_queue.cc:260] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.045073 9643 tablet_bootstrap.cc:492] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:12.045703 9643 ts_tablet_manager.cc:1403] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.196s user 0.013s sys 0.011s
I20260503 13:58:12.045558 9350 catalog_manager.cc:5671] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 1 to 2, leader changed from 4bea375a55034750a1f8804b0db97868 (127.8.26.195) to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 2 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:12.046583 9642 ts_tablet_manager.cc:1434] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.034s user 0.008s sys 0.020s
I20260503 13:58:12.047475 9642 tablet_bootstrap.cc:492] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:12.047768 9643 raft_consensus.cc:359] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.048331 9643 raft_consensus.cc:385] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.048593 9643 raft_consensus.cc:740] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.049178 9643 consensus_queue.cc:260] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.053227 9643 ts_tablet_manager.cc:1434] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260503 13:58:12.054558 9643 tablet_bootstrap.cc:492] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:12.062453 9644 tablet_bootstrap.cc:654] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.027871 9847 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.075451 9597 raft_consensus.cc:1275] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:12.077286 9643 tablet_bootstrap.cc:654] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.083981 9642 tablet_bootstrap.cc:654] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.090528 9845 consensus_queue.cc:1048] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:12.092049 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:12.102479 9847 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.105677 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:12.106240 9601 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:58:12.112778 9418 leader_election.cc:343] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.129827 9644 tablet_bootstrap.cc:492] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:12.130224 9644 ts_tablet_manager.cc:1403] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.125s user 0.008s sys 0.010s
I20260503 13:58:12.133237 9599 raft_consensus.cc:1275] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:12.134825 9839 consensus_queue.cc:1048] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:12.137557 9601 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:12.139384 9416 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:12.140120 9839 raft_consensus.cc:2804] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:12.140576 9839 raft_consensus.cc:697] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:12.141326 9839 consensus_queue.cc:237] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.147976 9644 raft_consensus.cc:359] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.148636 9644 raft_consensus.cc:385] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.148898 9644 raft_consensus.cc:740] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.152225 9644 consensus_queue.cc:260] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.157456 9644 ts_tablet_manager.cc:1434] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20260503 13:58:12.158308 9644 tablet_bootstrap.cc:492] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:12.159502 9643 tablet_bootstrap.cc:492] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:12.160141 9351 catalog_manager.cc:5671] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 1 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:12.162380 9643 ts_tablet_manager.cc:1403] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.108s user 0.005s sys 0.005s
I20260503 13:58:12.163473 9644 tablet_bootstrap.cc:654] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.161221 9642 tablet_bootstrap.cc:492] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:12.164883 9642 ts_tablet_manager.cc:1403] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.118s user 0.008s sys 0.003s
I20260503 13:58:12.167865 9642 raft_consensus.cc:359] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.168349 9642 raft_consensus.cc:385] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.168534 9642 raft_consensus.cc:740] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.169112 9642 consensus_queue.cc:260] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.169262 9643 raft_consensus.cc:359] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.169842 9643 raft_consensus.cc:385] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.170035 9643 raft_consensus.cc:740] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.170420 9643 consensus_queue.cc:260] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.173729 9642 ts_tablet_manager.cc:1434] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20260503 13:58:12.174505 9642 tablet_bootstrap.cc:492] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:12.178020 9643 ts_tablet_manager.cc:1434] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.015s user 0.003s sys 0.002s
I20260503 13:58:12.178996 9643 tablet_bootstrap.cc:492] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:12.192649 9643 tablet_bootstrap.cc:654] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.199354 9642 tablet_bootstrap.cc:654] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.201493 9599 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.206811 9525 raft_consensus.cc:3060] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.214701 9644 tablet_bootstrap.cc:492] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:12.215227 9644 ts_tablet_manager.cc:1403] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.009s
I20260503 13:58:12.218475 9644 raft_consensus.cc:359] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.219117 9644 raft_consensus.cc:385] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.219365 9644 raft_consensus.cc:740] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.219437 9642 tablet_bootstrap.cc:492] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:12.220368 9642 ts_tablet_manager.cc:1403] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.046s user 0.012s sys 0.008s
I20260503 13:58:12.219937 9644 consensus_queue.cc:260] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.224524 9644 ts_tablet_manager.cc:1434] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260503 13:58:12.225382 9644 tablet_bootstrap.cc:492] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:12.227449 9642 raft_consensus.cc:359] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.228097 9642 raft_consensus.cc:385] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.228348 9642 raft_consensus.cc:740] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.228951 9642 consensus_queue.cc:260] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.230602 9526 raft_consensus.cc:1275] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:12.230873 9642 ts_tablet_manager.cc:1434] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20260503 13:58:12.231675 9642 tablet_bootstrap.cc:492] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:12.232136 9839 consensus_queue.cc:1048] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:12.240104 9644 tablet_bootstrap.cc:654] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.252064 9839 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.252584 9839 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.256263 9839 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.256879 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:12.257581 9519 raft_consensus.cc:2468] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:12.258504 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:12.260074 9416 leader_election.cc:343] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.262128 9418 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:12.263129 9839 raft_consensus.cc:2804] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:12.263459 9839 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:12.263823 9839 raft_consensus.cc:3060] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.267707 9642 tablet_bootstrap.cc:654] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.270558 9839 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.273309 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:12.273874 9519 raft_consensus.cc:3060] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.275453 9845 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.275951 9845 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.277835 9845 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.279507 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:12.281163 9519 raft_consensus.cc:2468] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:12.282339 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:12.282624 9418 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:12.281059 9416 leader_election.cc:343] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:12.284222 9418 leader_election.cc:343] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.285244 9418 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:12.284775 9845 raft_consensus.cc:2804] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:12.286085 9838 raft_consensus.cc:2749] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:12.286557 9845 raft_consensus.cc:697] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:12.287402 9845 consensus_queue.cc:237] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.291805 9597 raft_consensus.cc:1275] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:12.293545 9847 consensus_queue.cc:1048] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:12.297195 9644 tablet_bootstrap.cc:492] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:12.297297 9839 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.297679 9644 ts_tablet_manager.cc:1403] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.006s
I20260503 13:58:12.299032 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:12.300267 9644 raft_consensus.cc:359] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.301055 9644 raft_consensus.cc:385] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.301301 9644 raft_consensus.cc:740] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.301865 9644 consensus_queue.cc:260] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:12.300315 9416 leader_election.cc:343] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.303260 9644 ts_tablet_manager.cc:1434] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260503 13:58:12.304177 9644 tablet_bootstrap.cc:492] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:12.305805 9643 tablet_bootstrap.cc:492] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:12.306213 9643 ts_tablet_manager.cc:1403] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.127s user 0.017s sys 0.015s
I20260503 13:58:12.308676 9643 raft_consensus.cc:359] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.309278 9643 raft_consensus.cc:385] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.311168 9643 raft_consensus.cc:740] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.311791 9643 consensus_queue.cc:260] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.313694 9643 ts_tablet_manager.cc:1434] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260503 13:58:12.314473 9643 tablet_bootstrap.cc:492] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:12.319854 9642 tablet_bootstrap.cc:492] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:12.320284 9642 ts_tablet_manager.cc:1403] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.006s
I20260503 13:58:12.322669 9642 raft_consensus.cc:359] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.323280 9642 raft_consensus.cc:385] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.323530 9642 raft_consensus.cc:740] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.324111 9856 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.324625 9856 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.326407 9856 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:12.327100 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:12.327666 9451 raft_consensus.cc:2468] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:12.324108 9642 consensus_queue.cc:260] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.328924 9566 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:12.329734 9856 raft_consensus.cc:2804] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:12.329897 9642 ts_tablet_manager.cc:1434] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260503 13:58:12.330224 9856 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:12.330641 9856 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.330813 9642 tablet_bootstrap.cc:492] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:12.333637 9644 tablet_bootstrap.cc:654] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.333730 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:12.335429 9567 leader_election.cc:343] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.336416 9642 tablet_bootstrap.cc:654] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.336572 9858 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.337011 9858 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.338768 9858 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:12.340216 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:12.340881 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:12.340265 9351 catalog_manager.cc:5671] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 1 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
W20260503 13:58:12.341776 9567 leader_election.cc:343] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:12.342257 9566 leader_election.cc:343] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.342629 9566 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:12.343498 9858 raft_consensus.cc:2749] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:12.346935 9643 tablet_bootstrap.cc:654] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.355777 9856 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.362445 9898 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.363262 9644 tablet_bootstrap.cc:492] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:12.363654 9644 ts_tablet_manager.cc:1403] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.060s user 0.013s sys 0.002s
I20260503 13:58:12.362905 9898 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.365271 9642 tablet_bootstrap.cc:492] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:12.365716 9642 ts_tablet_manager.cc:1403] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.005s
I20260503 13:58:12.367676 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:12.368471 9642 raft_consensus.cc:359] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.369163 9642 raft_consensus.cc:385] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.369447 9642 raft_consensus.cc:740] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.369561 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:12.370178 9451 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:12.370121 9642 consensus_queue.cc:260] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:12.371567 9491 leader_election.cc:343] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.372160 9644 raft_consensus.cc:359] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.372804 9644 raft_consensus.cc:385] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.372308 9492 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:12.373090 9644 raft_consensus.cc:740] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.373747 9725 raft_consensus.cc:2804] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:12.366232 9898 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.374188 9725 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:12.374581 9642 ts_tablet_manager.cc:1434] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260503 13:58:12.374652 9725 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.375537 9642 tablet_bootstrap.cc:492] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:12.377074 9644 consensus_queue.cc:260] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.377856 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:12.378396 9451 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.379050 9644 ts_tablet_manager.cc:1434] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.015s user 0.002s sys 0.003s
I20260503 13:58:12.381995 9643 tablet_bootstrap.cc:492] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:12.382412 9643 ts_tablet_manager.cc:1403] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.068s user 0.009s sys 0.006s
I20260503 13:58:12.383356 9644 tablet_bootstrap.cc:492] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:12.383337 9725 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.389937 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:12.390506 9450 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.390620 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:12.392055 9491 leader_election.cc:343] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.396795 9643 raft_consensus.cc:359] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.402437 9643 raft_consensus.cc:385] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.402763 9643 raft_consensus.cc:740] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.399122 9451 raft_consensus.cc:2468] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:12.400467 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:12.400893 9856 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:12.402000 9450 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:12.402570 9850 consensus_queue.cc:1048] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260503 13:58:12.404843 9525 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.408100 9492 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:12.408957 9898 raft_consensus.cc:2804] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:12.409381 9898 raft_consensus.cc:697] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:12.410123 9898 consensus_queue.cc:237] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.412127 9850 consensus_queue.cc:1048] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:12.413753 9644 tablet_bootstrap.cc:654] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.415463 9845 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.386013 9725 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.415953 9845 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.428090 9525 raft_consensus.cc:2468] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:12.428696 9845 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.429147 9567 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:12.430958 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:12.431671 9519 raft_consensus.cc:2468] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:12.433141 9850 raft_consensus.cc:2804] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:12.430861 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:12.433106 9418 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:12.434459 9845 raft_consensus.cc:2804] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260503 13:58:12.434743 9416 leader_election.cc:343] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.434832 9845 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:12.435387 9845 raft_consensus.cc:3060] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.435796 9850 raft_consensus.cc:697] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:12.436667 9850 consensus_queue.cc:237] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.426024 9643 consensus_queue.cc:260] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.439920 9642 tablet_bootstrap.cc:654] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.447564 9643 ts_tablet_manager.cc:1434] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.065s user 0.006s sys 0.001s
I20260503 13:58:12.448611 9643 tablet_bootstrap.cc:492] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:12.462363 9845 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.469429 9351 catalog_manager.cc:5671] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:12.472208 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:12.472863 9526 raft_consensus.cc:3060] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.469214 9856 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.474691 9856 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.480808 9856 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:12.482754 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:12.483371 9451 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:12.482750 9642 tablet_bootstrap.cc:492] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:12.491006 9642 ts_tablet_manager.cc:1403] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.116s user 0.012s sys 0.032s
I20260503 13:58:12.485524 9566 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:12.483982 9350 catalog_manager.cc:5671] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:12.493156 9856 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:12.494565 9856 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:12.494910 9856 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:12.487592 9643 tablet_bootstrap.cc:654] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.489457 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:12.498641 9642 raft_consensus.cc:359] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.499294 9642 raft_consensus.cc:385] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.498306 9525 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:12.502666 9725 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.503124 9725 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.499550 9642 raft_consensus.cc:740] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.508194 9642 consensus_queue.cc:260] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.513830 9725 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.497934 9845 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.513219 9642 ts_tablet_manager.cc:1434] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260503 13:58:12.518266 9642 tablet_bootstrap.cc:492] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:12.516875 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:12.519800 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:12.520620 9601 raft_consensus.cc:2393] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:12.532158 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:12.533361 9416 leader_election.cc:343] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.539299 9725 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.539520 9525 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.541324 9599 raft_consensus.cc:1275] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:12.543215 9725 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.545027 9725 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.545464 9725 consensus_queue.cc:1048] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:12.548126 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:12.565482 9491 leader_election.cc:343] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.584460 9856 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.586949 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:12.587498 9450 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 3
I20260503 13:58:12.588229 9847 consensus_queue.cc:1048] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:12.583369 9526 raft_consensus.cc:2468] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:12.591445 9418 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:12.593158 9847 raft_consensus.cc:2804] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:12.594435 9847 raft_consensus.cc:697] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:12.595168 9847 consensus_queue.cc:237] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.596119 9644 tablet_bootstrap.cc:492] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:12.596522 9644 ts_tablet_manager.cc:1403] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.214s user 0.030s sys 0.014s
I20260503 13:58:12.598917 9450 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:12.599852 9642 tablet_bootstrap.cc:654] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.600178 9566 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:12.599618 9859 consensus_queue.cc:1048] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260503 13:58:12.598942 9644 raft_consensus.cc:359] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.601563 9644 raft_consensus.cc:385] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.601811 9644 raft_consensus.cc:740] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.602599 9864 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:12.603891 9859 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.604585 9859 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.609514 9864 raft_consensus.cc:697] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:12.610060 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:12.610319 9864 consensus_queue.cc:237] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.615027 9644 consensus_queue.cc:260] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:12.616881 9418 leader_election.cc:343] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.617189 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:12.617648 9859 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.618256 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:12.617765 9597 raft_consensus.cc:2468] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:12.619268 9644 ts_tablet_manager.cc:1434] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.022s user 0.004s sys 0.001s
I20260503 13:58:12.602320 9856 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:12.626068 9644 tablet_bootstrap.cc:492] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:12.626451 9416 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:12.627408 9839 raft_consensus.cc:2804] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:12.627763 9526 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:12.608768 9492 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:12.627764 9839 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:12.630257 9839 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.632174 9643 tablet_bootstrap.cc:492] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:12.632663 9643 ts_tablet_manager.cc:1403] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.184s user 0.013s sys 0.003s
I20260503 13:58:12.641726 9349 catalog_manager.cc:5671] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 1 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:12.645466 9845 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.645820 9643 raft_consensus.cc:359] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.645977 9845 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.646476 9643 raft_consensus.cc:385] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.646749 9643 raft_consensus.cc:740] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.647404 9643 consensus_queue.cc:260] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.647943 9845 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.648864 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:12.649501 9643 ts_tablet_manager.cc:1434] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260503 13:58:12.649549 9519 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:12.650601 9643 tablet_bootstrap.cc:492] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:12.651014 9418 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:12.644340 9451 raft_consensus.cc:1275] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:12.653923 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:12.654510 9599 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:12.654937 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:12.656764 9950 consensus_queue.cc:1048] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:12.658037 9526 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:12.661064 9838 raft_consensus.cc:2804] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:12.661437 9838 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260503 13:58:12.657915 9492 leader_election.cc:343] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.663985 9838 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.658349 9839 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.678742 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:12.679292 9599 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.667611 9492 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:12.681401 9951 raft_consensus.cc:2749] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:12.681808 9950 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.683751 9839 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.689680 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:12.696224 9838 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.689097 9950 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.700703 9898 raft_consensus.cc:2749] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260503 13:58:12.692099 9418 leader_election.cc:343] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.705665 9642 tablet_bootstrap.cc:492] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:12.704551 9599 raft_consensus.cc:2468] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:12.706405 9642 ts_tablet_manager.cc:1403] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.189s user 0.016s sys 0.008s
I20260503 13:58:12.707828 9416 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:12.708451 9959 raft_consensus.cc:2804] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:12.709110 9959 raft_consensus.cc:697] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:12.709151 9838 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.710067 9959 consensus_queue.cc:237] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.710705 9642 raft_consensus.cc:359] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.711408 9642 raft_consensus.cc:385] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.711665 9642 raft_consensus.cc:740] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.712352 9847 consensus_queue.cc:1048] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:12.712620 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:12.713145 9519 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.713968 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:12.714496 9599 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.715634 9644 tablet_bootstrap.cc:654] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.712232 9642 consensus_queue.cc:260] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.700894 9351 catalog_manager.cc:5671] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:12.721299 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:12.724669 9642 ts_tablet_manager.cc:1434] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260503 13:58:12.725549 9642 tablet_bootstrap.cc:492] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
W20260503 13:58:12.726177 9492 leader_election.cc:343] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.726933 9950 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.729748 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:12.730775 9491 leader_election.cc:343] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:12.731182 9491 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:12.734776 9898 raft_consensus.cc:2749] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:12.749775 9643 tablet_bootstrap.cc:654] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.746454 9859 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:12.750463 9859 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.752642 9859 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.752755 9599 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:12.756588 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:12.757146 9601 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:12.759110 9416 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:12.759949 9416 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:12.760639 9839 raft_consensus.cc:2804] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:12.761611 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:12.761973 9838 raft_consensus.cc:2804] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:12.762348 9838 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:12.762696 9838 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:12.764290 9644 tablet_bootstrap.cc:492] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:12.764801 9644 ts_tablet_manager.cc:1403] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.139s user 0.006s sys 0.008s
I20260503 13:58:12.765273 9642 tablet_bootstrap.cc:654] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.765730 9519 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:12.761669 9839 raft_consensus.cc:697] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:12.766426 9525 raft_consensus.cc:2393] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 2: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:12.769322 9839 consensus_queue.cc:237] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.761386 9350 catalog_manager.cc:5671] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 1 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:12.781844 9858 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.782354 9858 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.784271 9858 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:12.784607 9416 consensus_peers.cc:597] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 -> Peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Couldn't send request to peer 4bea375a55034750a1f8804b0db97868. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260503 13:58:12.785532 9838 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.786929 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:12.788875 9451 raft_consensus.cc:2410] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260503 13:58:12.787119 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:12.790004 9519 raft_consensus.cc:2410] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260503 13:58:12.791081 9567 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:12.791567 9644 raft_consensus.cc:359] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.792032 9858 raft_consensus.cc:2749] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:12.792548 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:12.793299 9950 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.793730 9950 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.793767 9644 raft_consensus.cc:385] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.798125 9644 raft_consensus.cc:740] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.798724 9644 consensus_queue.cc:260] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.800694 9644 ts_tablet_manager.cc:1434] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.033s user 0.005s sys 0.001s
I20260503 13:58:12.801720 9644 tablet_bootstrap.cc:492] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:12.800673 9601 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:12.805168 9416 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:12.810808 9950 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.811991 9643 tablet_bootstrap.cc:492] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:12.812405 9643 ts_tablet_manager.cc:1403] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.162s user 0.011s sys 0.017s
I20260503 13:58:12.809306 9847 raft_consensus.cc:2804] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:12.812608 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:12.815524 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:12.816237 9525 raft_consensus.cc:2393] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 2: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:12.818310 9644 tablet_bootstrap.cc:654] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.808146 9838 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.818311 9599 raft_consensus.cc:2393] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:12.821446 9643 raft_consensus.cc:359] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.821935 9643 raft_consensus.cc:385] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:12.822125 9643 raft_consensus.cc:740] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:12.822799 9643 consensus_queue.cc:260] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.818913 9847 raft_consensus.cc:697] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:12.824137 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:12.826035 9847 consensus_queue.cc:237] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.829121 9643 ts_tablet_manager.cc:1434] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.016s user 0.000s sys 0.005s
I20260503 13:58:12.829384 9349 catalog_manager.cc:5671] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 1 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:12.839305 9856 consensus_queue.cc:1048] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:12.841490 9492 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:12.842706 9898 consensus_queue.cc:1048] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:12.843936 9898 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:12.846195 9642 tablet_bootstrap.cc:492] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:12.847081 9856 consensus_queue.cc:1048] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:12.850176 9642 ts_tablet_manager.cc:1403] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.125s user 0.011s sys 0.005s
I20260503 13:58:12.851905 9643 tablet_bootstrap.cc:492] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:12.868314 9724 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.875180 9724 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.885955 9864 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.888273 9864 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.890149 9864 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:12.890720 9643 tablet_bootstrap.cc:654] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:12.892287 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:12.892355 9724 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.892915 9451 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:12.906447 9725 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.944278 9850 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.944080 9725 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:12.948333 9850 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:13.695575 9566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.004s sys 0.017s
W20260503 13:58:13.705741 9566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.004s sys 0.017s
I20260503 13:58:13.722249 9644 tablet_bootstrap.cc:492] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:13.722813 9644 ts_tablet_manager.cc:1403] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.921s user 0.025s sys 0.014s
I20260503 13:58:13.693570 9642 raft_consensus.cc:359] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:13.729825 9642 raft_consensus.cc:385] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:13.730103 9642 raft_consensus.cc:740] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:13.730718 9642 consensus_queue.cc:260] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:13.666776 9898 consensus_meta.cc:287] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.822s user 0.004s sys 0.004s
I20260503 13:58:13.162535 9951 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:13.130098 9856 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:12.998198 9725 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:12.998379 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:12.998739 9850 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:13.962347 9969 log.cc:927] Time spent T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 1.054s user 0.000s sys 0.001s
I20260503 13:58:13.956151 9898 raft_consensus.cc:2749] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:13.967870 9850 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:13.969525 9864 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:13.967999 9724 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:13.974464 9970 log.cc:927] Time spent T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.724s user 0.001s sys 0.000s
I20260503 13:58:13.964818 9856 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:13.962953 9951 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:13.969054 9599 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
W20260503 13:58:14.014441 9968 log.cc:927] Time spent T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.831s user 0.001s sys 0.000s
I20260503 13:58:14.027441 9864 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.027007 9724 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.056274 9856 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.026398 9850 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.047719 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 119) took 1139 ms. Trace:
I20260503 13:58:14.069746 9858 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.072378 9858 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.066722 9856 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.070839 9976 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.089717 9644 raft_consensus.cc:359] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.090451 9644 raft_consensus.cc:385] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:14.090721 9644 raft_consensus.cc:740] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:14.091357 9644 consensus_queue.cc:260] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.069298 9599 rpcz_store.cc:276] 0503 13:58:12.943495 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:12.998081 (+ 54586us) service_pool.cc:224] Handling call
0503 13:58:14.040591 (+1042510us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:14.096115 9642 ts_tablet_manager.cc:1434] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 1.245s user 0.008s sys 0.024s
I20260503 13:58:14.086453 9858 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.092517 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 130) took 1243 ms. Trace:
W20260503 13:58:14.096371 9642 ts_tablet_manager.cc:1467] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a: Tablet startup took 1245ms
I20260503 13:58:14.084703 9856 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:14.099824 9642 ts_tablet_manager.cc:1469] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a: Trace:
0503 13:58:05.215136 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:05.494462 (+279326us) service_pool.cc:224] Handling call
0503 13:58:05.716542 (+222080us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:05.716670 (+ 128us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:05.770462 (+ 53792us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:05.870122 (+ 99660us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:05.871404 (+ 1282us) inbound_call.cc:177] Queueing success response
0503 13:58:12.725261 (+6853857us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0503 13:58:12.850941 (+125680us) ts_tablet_manager.cc:1435] Starting tablet replica
0503 13:58:12.891046 (+ 40105us) tablet_replica.cc:261] Starting instrumentation
0503 13:58:12.891488 (+ 442us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":175327,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":6421,"tablet-open.queue_time_us":6854757,"thread_start_us":244734,"threads_started":1}
I20260503 13:58:14.102499 9642 tablet_bootstrap.cc:492] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:14.102888 9644 ts_tablet_manager.cc:1434] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.380s user 0.004s sys 0.004s
I20260503 13:58:14.104156 9864 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.104960 9856 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.105768 9724 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.106454 9951 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.107117 9850 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.107402 9856 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:14.107831 9856 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.099695 9525 rpcz_store.cc:276] 0503 13:58:12.849130 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:12.849248 (+ 118us) service_pool.cc:224] Handling call
0503 13:58:12.849514 (+ 266us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:12.850268 (+ 754us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:12.850298 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:12.850707 (+ 409us) log.cc:844] Serialized 26 byte log entry
0503 13:58:12.880444 (+ 29737us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:12.881894 (+ 1450us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:12.883680 (+ 1786us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:14.092266 (+1208586us) raft_consensus.cc:1685] finished
0503 13:58:14.092291 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:14.092449 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":421,"raft.queue_time_us":1361,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":81,"thread_start_us":27026,"threads_started":1,"wal-append.queue_time_us":80028}
I20260503 13:58:14.093657 9976 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:14.114660 9976 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:14.116659 9856 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.141395 9644 tablet_bootstrap.cc:492] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:14.349550 9864 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.350085 9643 tablet_bootstrap.cc:492] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:14.350091 9864 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.350687 9643 ts_tablet_manager.cc:1403] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 1.499s user 0.016s sys 0.002s
I20260503 13:58:14.353637 9642 tablet_bootstrap.cc:654] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:14.354378 9856 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.355201 9856 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.355621 9864 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.356364 9976 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:14.358045 9976 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.359251 9856 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.359827 9976 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.361909 9644 tablet_bootstrap.cc:654] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:14.365274 9643 raft_consensus.cc:359] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.365885 9643 raft_consensus.cc:385] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:14.366135 9643 raft_consensus.cc:740] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:14.366698 9643 consensus_queue.cc:260] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.369063 9856 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.369518 9856 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.371348 9856 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.371888 9982 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.372051 9986 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.372370 9982 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.372458 9986 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.374341 9986 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.374354 9982 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.375146 9977 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.383319 9977 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.385280 9977 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.371768 9856 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.369038 9643 ts_tablet_manager.cc:1434] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260503 13:58:14.385855 9856 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.386518 9643 tablet_bootstrap.cc:492] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:14.387631 9856 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.390025 9991 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:14.390529 9991 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.375998 9976 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:14.375681 9864 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.376300 9978 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:14.376875 9972 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:14.377027 9850 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:14.376735 9984 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:13.693570 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.392475 9643 tablet_bootstrap.cc:654] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:14.415040 9850 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.392306 9644 tablet_bootstrap.cc:492] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:14.415736 9972 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.417168 9644 ts_tablet_manager.cc:1403] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.276s user 0.020s sys 0.004s
I20260503 13:58:14.417514 9864 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.416747 9978 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.392369 9991 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.421407 9864 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.419962 9976 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.424966 9526 raft_consensus.cc:2393] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:14.425505 9850 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.425470 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 133) took 1060 ms. Trace:
I20260503 13:58:14.426705 9526 rpcz_store.cc:276] 0503 13:58:13.365012 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:13.385299 (+ 20287us) service_pool.cc:224] Handling call
0503 13:58:14.425386 (+1040087us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:14.426705 9644 raft_consensus.cc:359] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.428206 9644 raft_consensus.cc:385] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:14.428500 9644 raft_consensus.cc:740] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:14.429172 9644 consensus_queue.cc:260] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.425029 9984 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.425973 9972 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.429097 9989 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.434840 9450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 135) took 1568 ms. Trace:
I20260503 13:58:14.434716 9989 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.435292 9450 rpcz_store.cc:276] 0503 13:58:12.866237 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:12.866373 (+ 136us) service_pool.cc:224] Handling call
0503 13:58:12.867203 (+ 830us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:12.867502 (+ 299us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:12.867531 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:12.867924 (+ 393us) log.cc:844] Serialized 26 byte log entry
0503 13:58:12.887602 (+ 19678us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:12.895185 (+ 7583us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:13.058259 (+163074us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:14.434625 (+1376366us) raft_consensus.cc:1685] finished
0503 13:58:14.434648 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:14.434783 (+ 135us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1270,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":6035,"spinlock_wait_cycles":40064,"thread_start_us":15111,"threads_started":1,"wal-append.queue_time_us":1164463}
I20260503 13:58:14.439882 9989 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.442400 9850 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.445493 9850 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.447986 9972 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.448441 9972 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:14.443569 9494 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.130s user 0.007s sys 0.011s
W20260503 13:58:14.457273 9494 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.160s user 0.009s sys 0.012s
I20260503 13:58:14.457903 9950 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:14.458141 9979 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.458380 9950 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.458642 9979 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.460156 9950 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.460589 9979 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.460722 9951 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:14.461082 9979 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.461205 9951 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.461516 9979 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.462919 9951 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.463249 9979 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.463668 9950 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.464257 9950 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.466192 9950 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.466682 9978 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.467535 9976 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:12.943264 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.473996 9642 tablet_bootstrap.cc:492] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:14.474311 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 128) took 1633 ms. Trace:
I20260503 13:58:14.474459 9642 ts_tablet_manager.cc:1403] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.372s user 0.008s sys 0.010s
I20260503 13:58:14.474661 9445 rpcz_store.cc:276] 0503 13:58:12.841013 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:12.841222 (+ 209us) service_pool.cc:224] Handling call
0503 13:58:12.841522 (+ 300us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:12.841839 (+ 317us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:12.841867 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:12.842236 (+ 369us) log.cc:844] Serialized 26 byte log entry
0503 13:58:12.878031 (+ 35795us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:12.878428 (+ 397us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:12.880857 (+ 2429us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:14.474081 (+1593224us) raft_consensus.cc:1685] finished
0503 13:58:14.474112 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:14.474258 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":913,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":81,"thread_start_us":31096,"threads_started":1,"wal-append.queue_time_us":55259}
W20260503 13:58:14.477357 9493 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.112s user 0.010s sys 0.005s
W20260503 13:58:14.477622 9493 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.112s user 0.011s sys 0.005s
I20260503 13:58:14.478615 9950 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:14.479066 9950 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.480053 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.480625 9526 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:14.480804 9950 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.481105 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.481284 9950 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.481670 9524 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:14.481724 9950 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.482231 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.476999 9642 raft_consensus.cc:359] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.482735 9642 raft_consensus.cc:385] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:14.483063 9642 raft_consensus.cc:740] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:14.483533 9974 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.483644 9642 consensus_queue.cc:260] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.484014 9974 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.486006 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.486378 9974 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.484752 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.487047 9525 raft_consensus.cc:2468] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:14.487037 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.488091 9519 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:14.482774 9527 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:14.488921 9525 raft_consensus.cc:1275] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260503 13:58:12.943578 9491 consensus_peers.cc:597] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a -> Peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Couldn't send request to peer 4bea375a55034750a1f8804b0db97868. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260503 13:58:14.490118 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 1.58395e+06us
I20260503 13:58:14.491207 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.491519 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:14.491828 9523 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:14.492944 9524 raft_consensus.cc:2468] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:14.493566 9950 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.493649 9975 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.494226 9599 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:14.494083 9950 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:14.494153 9975 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.494983 9950 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.496146 9975 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.496753 9950 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.497658 9642 ts_tablet_manager.cc:1434] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20260503 13:58:14.498517 9642 tablet_bootstrap.cc:492] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:14.498433 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:14.499605 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:14.500159 9598 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:14.501048 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.501655 9523 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:14.504293 9999 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:14.506243 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.506815 9524 raft_consensus.cc:2468] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:14.509665 9972 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.510167 9521 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
W20260503 13:58:14.510685 9521 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 132) took 1613 ms (client timeout 1930 ms). Trace:
I20260503 13:58:14.512776 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.511126 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.477613 9995 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:14.518392 9999 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.531250 9995 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.532363 9999 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.527200 9491 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.530666 9523 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
W20260503 13:58:14.511051 9521 rpcz_store.cc:269] 0503 13:58:12.897316 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:12.901144 (+ 3828us) service_pool.cc:224] Handling call
0503 13:58:14.510629 (+1609485us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:14.527016 9979 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.537523 9979 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.540040 9979 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:14.528340 9492 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.162s user 0.029s sys 0.035s
W20260503 13:58:14.543076 9492 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.230s user 0.030s sys 0.036s
I20260503 13:58:14.529309 9526 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:14.537113 9974 raft_consensus.cc:2804] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:14.545617 9974 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:14.545970 9974 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:14.533910 9491 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.552276 9975 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.570237 9898 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.570693 10006 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:14.571142 9898 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.572798 9898 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.573195 9898 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:14.573427 9898 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:14.573670 9898 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:14.574510 9351 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51410 (request call id 62) took 1720 ms. Trace:
I20260503 13:58:14.574883 9351 rpcz_store.cc:276] 0503 13:58:12.854292 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:12.854404 (+ 112us) service_pool.cc:224] Handling call
0503 13:58:14.574442 (+1720038us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":1}
I20260503 13:58:14.562613 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.586697 10001 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.587209 10001 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.589233 10001 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.591053 10005 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.592250 10005 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.502127 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:14.594702 9601 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:14.595471 10005 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.596007 10005 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:14.596451 10005 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.598361 10005 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.501060 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:14.617775 9993 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:14.620316 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.620797 9999 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.621322 9999 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.623288 9999 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.627264 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.627924 9448 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:14.628834 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.631440 9451 raft_consensus.cc:1275] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:14.632486 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.634770 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.635964 9448 raft_consensus.cc:2468] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:14.637027 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.637800 9452 raft_consensus.cc:2393] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:14.638670 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.640321 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.641068 9451 raft_consensus.cc:2393] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:14.644164 9989 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:14.644694 9989 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.644182 9850 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.618387 9993 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.648698 9973 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.649322 9973 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.650573 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.652948 10003 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.576134 10006 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.653584 9973 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.653391 10003 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.645807 9491 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.565646 9975 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.648680 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.647686 9523 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:14.671912 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:14.698086 8304 debug-util.cc:398] Leaking SignalData structure 0x7b080000fc40 after lost signal to thread 9492
I20260503 13:58:14.648794 9997 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:14.753825 9451 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:14.647826 9995 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.731711 9989 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.715904 9898 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:14.745118 9492 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.26.193:45599 timed out after 1.566s (SENT) [suppressed 4 similar messages]
I20260503 13:58:14.672353 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.731352 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.762259 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.714852 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.734469 9993 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.734215 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.713903 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.731758 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.667438 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.738898 9974 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.776103 10006 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.777869 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.779392 9643 tablet_bootstrap.cc:492] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:14.757659 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.774971 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.756322 9525 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:14.783562 9524 raft_consensus.cc:2468] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:14.782270 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.783761 9986 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:14.787647 9643 ts_tablet_manager.cc:1403] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.401s user 0.017s sys 0.002s
I20260503 13:58:14.733646 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.772447 9973 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:14.787442 9453 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
W20260503 13:58:14.791859 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 46777.5us
I20260503 13:58:14.792294 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:14.765918 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 181532us
I20260503 13:58:14.784159 9452 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
W20260503 13:58:14.794590 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:14.793608 9973 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.798491 9973 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:14.798635 9985 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.798870 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.793606 9986 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.799978 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.800913 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.798940 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.801635 9986 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.800809 9643 raft_consensus.cc:359] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.802711 9643 raft_consensus.cc:385] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:14.802956 9643 raft_consensus.cc:740] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:14.801667 9451 raft_consensus.cc:2393] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:14.788714 9446 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:14.805398 10003 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.805578 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.802520 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.768656 9997 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.790804 10005 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.803828 9643 consensus_queue.cc:260] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.806806 9446 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:14.802296 9975 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.798583 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.802363 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.803620 9898 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.790921 9520 raft_consensus.cc:2468] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:14.790220 9519 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:14.817026 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.823149 9643 ts_tablet_manager.cc:1434] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20260503 13:58:14.850581 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:14.852142 9643 tablet_bootstrap.cc:492] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:14.852468 9599 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:14.817935 9984 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.819083 9567 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.809376 9973 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.854758 9644 ts_tablet_manager.cc:1434] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.437s user 0.017s sys 0.029s
I20260503 13:58:14.854959 9973 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.855630 9644 tablet_bootstrap.cc:492] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:14.856094 9527 raft_consensus.cc:2393] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 2: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:14.820147 10005 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.804337 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.812286 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.859862 9453 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:14.860756 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.861761 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:14.862249 9453 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:14.862484 9492 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:14.863687 10005 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.864327 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:14.864643 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 45648.3us
I20260503 13:58:14.865177 9973 raft_consensus.cc:2804] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:14.865520 9973 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:14.866320 9973 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:14.823659 9524 raft_consensus.cc:2468] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:14.864234 9492 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:14.868054 9984 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:14.868431 9984 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:14.868768 9984 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:14.868675 9567 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.869175 9898 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:14.869557 9567 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.870406 9567 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.870726 9978 raft_consensus.cc:2804] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:14.871104 9978 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:14.871410 9978 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:14.871622 9567 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.872509 9977 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:14.823091 9974 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.872862 9977 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:14.873665 9977 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:14.873797 9997 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.874570 9864 raft_consensus.cc:2804] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:14.869565 9898 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:14.875351 9850 raft_consensus.cc:2804] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:14.876886 9850 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.876755 9898 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:14.877303 9850 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:14.878640 9972 consensus_queue.cc:1048] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260503 13:58:14.875766 9864 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.880280 9864 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:14.883149 9972 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.883602 9972 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.887794 9972 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.821949 9985 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.894941 9898 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.895409 9973 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.896348 9850 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.897115 9898 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.897529 9973 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.898598 9850 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.899518 9973 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:14.899960 9973 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.902341 9973 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.888743 9567 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.904747 9864 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:14.905187 9978 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.906834 9978 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.905484 9567 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.907330 9978 raft_consensus.cc:2804] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:14.907658 9978 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.908016 9978 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:14.908279 9972 raft_consensus.cc:2804] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:14.908656 9972 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:14.908987 9972 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:14.907820 9567 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.911206 9976 raft_consensus.cc:2804] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:14.909456 9643 tablet_bootstrap.cc:654] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:14.911537 9976 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.911895 9976 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:14.912357 9567 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.915426 9567 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.921383 9978 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.922372 9567 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.923660 9978 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.865477 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.925551 9972 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.917527 9864 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:14.910327 9984 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.924216 9978 raft_consensus.cc:2804] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:14.947002 9972 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.947166 9978 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:14.947532 9978 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:14.918550 9977 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.925101 9644 tablet_bootstrap.cc:654] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:14.930629 9643 tablet_bootstrap.cc:492] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:14.948351 9643 ts_tablet_manager.cc:1403] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.096s user 0.011s sys 0.004s
I20260503 13:58:14.949326 9977 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.925598 9567 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.950333 9567 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.950794 9643 raft_consensus.cc:359] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.951151 9567 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.951454 9643 raft_consensus.cc:385] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:14.951730 9643 raft_consensus.cc:740] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:14.952708 9972 raft_consensus.cc:2804] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:14.952418 9643 consensus_queue.cc:260] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.953049 9972 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:14.953402 9972 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:14.953555 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.953778 9978 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.955802 9978 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.956511 9977 raft_consensus.cc:2804] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:14.956872 9977 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:14.952451 9567 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.957472 9643 ts_tablet_manager.cc:1434] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260503 13:58:14.957810 9864 raft_consensus.cc:2804] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:14.958316 9643 tablet_bootstrap.cc:492] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:14.957074 9984 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.958164 9864 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:14.959050 9864 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:14.961899 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.962415 9451 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:14.957877 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.956267 9978 raft_consensus.cc:2804] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:14.963367 9526 raft_consensus.cc:2468] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:14.960462 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:14.965829 9643 tablet_bootstrap.cc:654] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:14.966121 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:14.966622 9451 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:14.958127 9977 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:14.967844 9864 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.957926 9567 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.969153 9984 raft_consensus.cc:2804] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:14.969293 9567 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:14.971557 9451 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:14.925283 10014 raft_consensus.cc:2804] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:14.944372 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:14.969638 9864 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.827943 9445 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:14.918478 9850 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:14.917744 9976 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.973572 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.972535 10014 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:14.975220 9850 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.977056 9850 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:14.963392 9978 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:14.972524 9984 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:14.981063 9984 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:14.976964 10014 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:14.981914 9976 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:14.986435 9492 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:14.988508 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:14.989678 9984 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.009114 9984 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:15.009727 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.010159 9644 tablet_bootstrap.cc:492] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:15.010488 9492 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:15.011087 9644 ts_tablet_manager.cc:1403] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.156s user 0.016s sys 0.004s
I20260503 13:58:15.011615 9566 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:14.982136 9978 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:14.993091 9985 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:14.994112 9976 raft_consensus.cc:2804] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:15.015661 9898 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:15.017252 9898 raft_consensus.cc:697] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:14.994464 9973 raft_consensus.cc:2804] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:15.018069 9898 consensus_queue.cc:237] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.014261 9991 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:15.019496 9991 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:14.996276 9850 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.022222 9991 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:15.021320 9973 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:15.024698 9973 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
W20260503 13:58:15.012895 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 43683.3us
I20260503 13:58:14.995831 9972 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.026995 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.016456 9976 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:15.029825 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.031098 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.031672 9448 raft_consensus.cc:2468] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:15.034850 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.035019 9492 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:15.035364 9451 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.036715 9974 raft_consensus.cc:2804] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:15.037076 9974 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:15.037381 9974 raft_consensus.cc:3060] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:15.037827 9975 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:15.038316 9975 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.039129 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.024667 9984 raft_consensus.cc:3060] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:15.045446 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.046084 9452 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:15.015743 9977 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.047868 9492 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:15.045310 9644 raft_consensus.cc:359] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.049041 9644 raft_consensus.cc:385] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:15.049325 9644 raft_consensus.cc:740] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:15.027661 9850 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:15.050076 9644 consensus_queue.cc:260] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.034545 9976 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:15.052542 9644 ts_tablet_manager.cc:1434] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20260503 13:58:15.059942 9644 tablet_bootstrap.cc:492] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:15.054229 9951 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:15.060953 9951 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:15.061281 9951 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:15.053565 9451 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:15.054263 10016 consensus_queue.cc:1048] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:15.054996 9978 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.067759 9976 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.068648 10017 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:15.057911 9975 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:15.069353 9984 raft_consensus.cc:2749] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:15.069870 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.070689 9972 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:15.071201 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.071632 9974 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.088006 9976 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:15.072880 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.089138 9974 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:15.089373 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.089972 9445 raft_consensus.cc:2393] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:15.048635 9567 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
W20260503 13:58:15.200222 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 156069us
I20260503 13:58:15.203470 9567 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:15.075237 10014 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.077239 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.084460 10017 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.204474 9864 raft_consensus.cc:2804] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:15.204842 9864 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:15.204986 9448 raft_consensus.cc:2393] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:15.205222 9864 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.205513 10014 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:15.205987 10014 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:15.206368 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.206576 10017 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:15.086555 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.207408 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.207731 9450 raft_consensus.cc:2393] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 3: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:15.087028 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.208650 9452 raft_consensus.cc:2393] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:15.083813 9453 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:15.090498 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.212968 9996 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:15.068363 9951 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.080250 9973 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.247671 9951 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:15.248307 9973 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:15.069808 9977 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:15.084371 9978 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:15.104997 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.283221 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.133419 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.153323 9642 tablet_bootstrap.cc:654] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:15.175474 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.177651 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.283869 9449 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:15.284821 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.285341 9520 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:15.178398 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.180389 9418 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:15.289060 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 108721us
I20260503 13:58:15.289450 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.183355 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.291070 9995 raft_consensus.cc:2804] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:15.291431 9995 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:15.291733 9995 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.179061 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.293025 9524 raft_consensus.cc:2393] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:15.293757 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.192505 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.296180 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.189960 9644 tablet_bootstrap.cc:654] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:15.297200 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.198334 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.299748 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:15.299768 9995 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.300292 9597 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.302677 9995 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:15.199440 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.201864 9975 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:15.091183 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.304960 9451 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.304864 9975 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.213423 9996 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.306720 9975 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:15.202370 10020 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:15.307317 10020 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.222097 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.229056 10014 raft_consensus.cc:2749] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:15.222043 9864 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.240859 9987 consensus_queue.cc:1048] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260503 13:58:15.241374 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.242344 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.243431 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.257091 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.258181 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.261819 9991 raft_consensus.cc:493] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:15.265173 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.266633 9593 raft_consensus.cc:1240] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:15.267616 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.312618 9596 raft_consensus.cc:2393] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:15.312829 9451 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:15.313434 9991 raft_consensus.cc:515] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.314697 9598 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:15.271054 9643 tablet_bootstrap.cc:492] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:15.271402 9491 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:15.272949 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.274894 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.316370 9447 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:15.316772 9600 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:15.317235 9523 raft_consensus.cc:2393] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 3: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:15.317420 9521 raft_consensus.cc:2393] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:15.278116 9599 raft_consensus.cc:1240] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:15.280066 9492 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:15.318678 9492 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:15.280457 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 220259us
I20260503 13:58:15.281932 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.282603 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.283674 9452 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.285265 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.322628 9527 raft_consensus.cc:2368] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3.
I20260503 13:58:15.194253 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.323484 9597 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:15.304472 9642 tablet_bootstrap.cc:492] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:15.307595 9996 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:15.230811 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.330247 9492 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:15.330369 9601 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:15.331117 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.331723 9601 raft_consensus.cc:2393] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:15.231923 10017 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:15.309963 9351 catalog_manager.cc:5671] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 1 to 2, leader changed from 4bea375a55034750a1f8804b0db97868 (127.8.26.195) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 2 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:15.315639 9643 ts_tablet_manager.cc:1403] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.358s user 0.023s sys 0.020s
I20260503 13:58:15.315164 9991 leader_election.cc:290] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:15.319343 9566 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:15.335194 9566 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:15.335961 9566 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:15.323374 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.324637 9973 raft_consensus.cc:2749] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:15.324905 9522 raft_consensus.cc:2393] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:15.325827 9642 ts_tablet_manager.cc:1403] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.827s user 0.011s sys 0.014s
I20260503 13:58:15.325052 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.324112 9453 raft_consensus.cc:3060] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:15.326763 9445 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:15.315878 10001 consensus_queue.cc:1048] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260503 13:58:15.325471 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.317209 9449 raft_consensus.cc:2468] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:15.328220 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.328810 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.313452 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:15.329500 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 58171.8us
I20260503 13:58:15.326622 9951 raft_consensus.cc:2804] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:15.332868 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.352946 9451 raft_consensus.cc:2468] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:15.337190 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.353790 9601 raft_consensus.cc:2393] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:15.328048 10020 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:15.338528 9566 leader_election.cc:343] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:15.320544 9975 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:15.308122 9520 raft_consensus.cc:2468] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:15.354118 9492 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:15.354453 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.328143 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.340377 9977 raft_consensus.cc:2804] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:15.341213 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:15.341485 9416 leader_election.cc:343] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:15.337847 10017 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.342401 9567 leader_election.cc:400] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Vote denied by peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805) with higher term. Message: Invalid argument: T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3.
I20260503 13:58:15.341739 9978 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:15.362013 9567 leader_election.cc:403] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260503 13:58:15.339349 10014 raft_consensus.cc:2749] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:15.349654 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.349239 9602 raft_consensus.cc:2468] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:15.353300 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:15.347087 9521 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:15.370661 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:15.370738 9491 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:15.360714 9418 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:15.358286 9566 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:15.364092 9567 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:15.344997 9595 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:15.355599 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:15.358265 9951 raft_consensus.cc:697] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:15.366151 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.363636 9600 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.367555 10017 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:15.367909 9598 raft_consensus.cc:2468] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:15.368135 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.368479 9593 raft_consensus.cc:2393] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:15.362437 9978 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:15.356783 9724 raft_consensus.cc:2804] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:15.367579 9643 raft_consensus.cc:359] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.382237 9643 raft_consensus.cc:385] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:15.382493 9643 raft_consensus.cc:740] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:15.344655 9528 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:15.358180 9977 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:15.386045 9974 raft_consensus.cc:2804] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:15.357093 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.383035 9643 consensus_queue.cc:260] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.366067 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.361640 9975 raft_consensus.cc:697] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:15.388911 9567 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:15.363762 9416 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:15.377130 9566 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:15.387130 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.377542 9642 raft_consensus.cc:359] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.375072 9950 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:15.374645 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.389864 9453 raft_consensus.cc:2468] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:15.387959 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:15.390336 9856 raft_consensus.cc:2804] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:15.389828 9642 raft_consensus.cc:385] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:15.364547 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.386085 9977 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:15.389468 9864 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:15.375519 9984 raft_consensus.cc:2804] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:15.392601 9566 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:15.389799 9522 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:15.380867 9601 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:15.374622 10014 raft_consensus.cc:2749] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:15.385273 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.382645 9724 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:15.390398 9523 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.390664 9594 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.388646 9643 ts_tablet_manager.cc:1434] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.055s user 0.003s sys 0.005s
I20260503 13:58:15.391117 9492 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:15.394428 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.394853 9991 raft_consensus.cc:2749] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:15.397614 9972 raft_consensus.cc:2804] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:15.382354 9978 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.399063 9724 raft_consensus.cc:3060] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:15.399549 9643 tablet_bootstrap.cc:492] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:15.400085 9973 raft_consensus.cc:2804] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:15.378369 10001 raft_consensus.cc:2749] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:15.396102 9566 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:15.396932 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.391016 9642 raft_consensus.cc:740] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:15.417783 9601 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:15.390069 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.390094 9976 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:15.398378 9644 tablet_bootstrap.cc:492] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:15.393210 9856 raft_consensus.cc:697] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
W20260503 13:58:15.397665 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 56218us
I20260503 13:58:15.421344 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.396771 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.434276 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.436664 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.437088 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.437227 9596 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:15.437363 9600 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:15.437650 9598 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:15.438092 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:15.438295 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.386300 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.424880 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.424393 10014 raft_consensus.cc:2804] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:15.400533 9973 raft_consensus.cc:697] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:15.438939 9600 raft_consensus.cc:2393] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:15.426036 9644 ts_tablet_manager.cc:1403] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.366s user 0.021s sys 0.013s
I20260503 13:58:15.426184 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:15.394227 9951 consensus_queue.cc:237] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.417682 9491 leader_election.cc:304] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:15.432585 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.432695 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.447484 9450 raft_consensus.cc:2393] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:15.447878 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.448261 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.448926 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.449349 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.417543 9984 raft_consensus.cc:697] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:15.431710 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.433259 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:15.433468 9418 leader_election.cc:343] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:15.388532 9975 consensus_queue.cc:237] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.433789 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.434338 9856 consensus_queue.cc:237] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.431833 9976 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.434882 9416 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:15.438786 9596 raft_consensus.cc:2393] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:15.441442 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.456693 9451 raft_consensus.cc:2393] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:15.393741 10016 raft_consensus.cc:2749] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:15.443419 9526 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:15.461127 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.461382 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.461881 9526 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.462486 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.462937 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.463515 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.418241 9642 consensus_queue.cc:260] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.426069 9972 raft_consensus.cc:697] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:15.464975 9974 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:15.438740 10020 raft_consensus.cc:2749] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805) with higher term. Message: Invalid argument: T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3.
I20260503 13:58:15.445643 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.426594 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.465387 9974 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.465878 9602 raft_consensus.cc:2393] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:15.466209 9950 raft_consensus.cc:2749] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:15.466877 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.467532 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.436664 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.467931 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.444787 9528 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.468452 9519 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:15.469862 9976 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:15.445307 9973 consensus_queue.cc:237] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.446772 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:15.453186 9898 raft_consensus.cc:2749] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:15.453205 9984 consensus_queue.cc:237] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:15.454170 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 58262.9us
I20260503 13:58:15.442078 9987 consensus_queue.cc:1059] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260503 13:58:15.438515 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.473107 9601 raft_consensus.cc:3060] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:15.446143 10014 raft_consensus.cc:697] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:15.474680 9567 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:15.483503 9567 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:15.445077 9523 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:15.472280 9593 raft_consensus.cc:1240] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:15.487762 9524 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:15.491039 9976 raft_consensus.cc:2804] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:15.490844 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:15.491403 9976 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:15.491768 9976 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.468601 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.492323 9593 raft_consensus.cc:1240] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:15.496876 9567 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:15.453626 9977 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.490813 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.491410 9525 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:15.502167 10020 raft_consensus.cc:2749] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:15.509965 9520 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:15.451130 9978 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:15.534255 9549 debug-util.cc:398] Leaking SignalData structure 0x7b08001670a0 after lost signal to thread 8302
I20260503 13:58:15.534500 9600 raft_consensus.cc:2393] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:15.478160 9991 raft_consensus.cc:2749] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:15.480105 9599 raft_consensus.cc:2376] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3. Re-sending same reply.
I20260503 13:58:15.546440 9597 raft_consensus.cc:2468] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
W20260503 13:58:15.494480 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 59679.6us
I20260503 13:58:15.535099 10016 raft_consensus.cc:2804] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:15.546193 9950 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:15.593325 9598 raft_consensus.cc:1240] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:15.593572 9950 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.595388 9950 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:15.581174 9447 raft_consensus.cc:1275] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:15.595836 9724 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.593225 9522 raft_consensus.cc:2468] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:15.597337 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.597734 9445 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:15.581431 9995 raft_consensus.cc:3055] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:15.598215 9995 raft_consensus.cc:740] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:15.598397 9644 raft_consensus.cc:359] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.599035 9644 raft_consensus.cc:385] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:15.599273 9644 raft_consensus.cc:740] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:15.599754 9527 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:15.600042 9521 raft_consensus.cc:2468] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:15.599809 9644 consensus_queue.cc:260] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.551062 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.601102 9595 raft_consensus.cc:2393] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:15.600258 9526 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:15.603344 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.603503 9995 consensus_queue.cc:260] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.535296 9972 consensus_queue.cc:237] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.604645 9995 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:15.605620 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.601066 9523 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.584359 9567 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:15.585168 9642 ts_tablet_manager.cc:1434] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.245s user 0.010s sys 0.011s
I20260503 13:58:15.505962 9491 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:15.592442 9528 raft_consensus.cc:2468] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:15.583561 9452 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:15.597820 9522 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.580999 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.608202 9642 tablet_bootstrap.cc:492] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
W20260503 13:58:15.612516 9492 leader_election.cc:343] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:15.613620 9643 tablet_bootstrap.cc:654] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:15.623842 9567 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:15.634991 9972 raft_consensus.cc:2804] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:15.635108 9567 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:15.635979 9567 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:15.632377 9523 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:15.636641 9858 raft_consensus.cc:2804] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:15.624996 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.640525 9567 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:15.641892 9858 raft_consensus.cc:697] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:15.642217 10036 raft_consensus.cc:2804] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:15.643203 9858 consensus_queue.cc:237] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.644155 10036 raft_consensus.cc:697] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:15.640519 10037 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:15.647814 9492 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:15.645074 10036 consensus_queue.cc:237] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.656786 9492 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:15.648442 9527 raft_consensus.cc:2393] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:15.606293 9977 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:15.603513 9519 raft_consensus.cc:2468] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:15.611995 10016 raft_consensus.cc:697] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:15.664716 9642 tablet_bootstrap.cc:654] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:15.671156 9644 ts_tablet_manager.cc:1434] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.226s user 0.005s sys 0.001s
I20260503 13:58:15.649148 10017 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:15.672163 9644 tablet_bootstrap.cc:492] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
W20260503 13:58:15.673429 9566 leader_election.cc:343] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:15.649159 9975 raft_consensus.cc:2749] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260503 13:58:15.663125 9549 debug-util.cc:398] Leaking SignalData structure 0x7b080016b780 after lost signal to thread 9385
I20260503 13:58:15.662057 9444 raft_consensus.cc:1240] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:15.617854 9976 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.621248 9950 consensus_queue.cc:1048] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260503 13:58:15.657696 10037 raft_consensus.cc:697] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:15.664086 9951 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:15.683522 9951 raft_consensus.cc:697] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:15.684494 9951 consensus_queue.cc:237] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.687497 10037 consensus_queue.cc:237] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.668860 9972 raft_consensus.cc:697] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:15.676290 9525 raft_consensus.cc:2393] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
W20260503 13:58:15.664887 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 167857us
I20260503 13:58:15.666590 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.677676 9594 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:15.691946 9567 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:15.676440 9978 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:15.682435 9973 raft_consensus.cc:2804] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:15.589074 10014 consensus_queue.cc:237] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:15.694202 9549 debug-util.cc:398] Leaking SignalData structure 0x7b0800285080 after lost signal to thread 9477
I20260503 13:58:15.679266 10017 raft_consensus.cc:697] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
W20260503 13:58:15.705170 10029 log.cc:927] Time spent T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260503 13:58:15.706595 10017 consensus_queue.cc:237] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.708730 9519 raft_consensus.cc:1240] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:15.710456 9597 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:58:15.710706 9549 debug-util.cc:398] Leaking SignalData structure 0x7b080024f320 after lost signal to thread 9552
I20260503 13:58:15.711453 9598 raft_consensus.cc:1240] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
W20260503 13:58:15.711727 9549 debug-util.cc:398] Leaking SignalData structure 0x7b080025a1c0 after lost signal to thread 9626
W20260503 13:58:15.712280 9549 debug-util.cc:398] Leaking SignalData structure 0x7b080017e880 after lost signal to thread 9946
I20260503 13:58:15.712589 9598 raft_consensus.cc:1240] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:15.690883 9600 raft_consensus.cc:1240] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:15.699776 9567 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:15.700773 9596 raft_consensus.cc:1240] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:15.650522 9416 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:15.701244 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.690678 9528 raft_consensus.cc:1240] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:15.694463 9724 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:15.704157 9977 raft_consensus.cc:2749] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:15.694535 9491 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:15.705174 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:15.697830 10025 log.cc:927] Time spent T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260503 13:58:15.715030 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.690590 9593 raft_consensus.cc:1240] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:15.715628 9567 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:15.736243 9567 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:15.737953 9567 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:15.718786 9978 raft_consensus.cc:2804] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:15.710439 9520 raft_consensus.cc:2468] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:15.696916 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:15.728309 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 105944us
I20260503 13:58:15.728619 9972 consensus_queue.cc:237] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.702075 9521 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.719211 9973 raft_consensus.cc:697] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:15.754397 9642 tablet_bootstrap.cc:492] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:15.754918 9642 ts_tablet_manager.cc:1403] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.147s user 0.019s sys 0.021s
I20260503 13:58:15.726284 9453 raft_consensus.cc:3060] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:15.723582 9643 tablet_bootstrap.cc:492] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:15.734781 9987 raft_consensus.cc:2764] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260503 13:58:15.759299 9643 ts_tablet_manager.cc:1403] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.360s user 0.024s sys 0.006s
I20260503 13:58:15.741389 9977 raft_consensus.cc:2749] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:15.742442 9864 raft_consensus.cc:2804] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:15.743294 10026 raft_consensus.cc:2749] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:15.752166 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.753692 10034 raft_consensus.cc:2749] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:15.775398 9520 raft_consensus.cc:1240] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:15.777866 9595 raft_consensus.cc:1240] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:15.756441 9978 raft_consensus.cc:697] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
W20260503 13:58:15.758059 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 61943.2us
I20260503 13:58:15.772701 9599 raft_consensus.cc:1240] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:15.698020 10016 consensus_queue.cc:237] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.780174 9864 raft_consensus.cc:697] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:15.811664 9642 raft_consensus.cc:359] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.812161 9642 raft_consensus.cc:385] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:15.812414 9642 raft_consensus.cc:740] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:15.813131 9642 consensus_queue.cc:260] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.802940 9976 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:15.797912 9973 consensus_queue.cc:237] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.773963 9416 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:15.859431 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:15.868667 9642 ts_tablet_manager.cc:1434] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.113s user 0.003s sys 0.003s
I20260503 13:58:15.807196 9864 consensus_queue.cc:237] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.869946 9520 raft_consensus.cc:1240] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:15.870743 9643 raft_consensus.cc:359] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.871515 9527 raft_consensus.cc:1240] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:15.871708 9643 raft_consensus.cc:385] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:15.872083 9520 raft_consensus.cc:1240] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:15.873876 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.872093 9643 raft_consensus.cc:740] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:15.882145 9643 consensus_queue.cc:260] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.888823 9519 raft_consensus.cc:1275] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:15.890501 9491 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:15.891320 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:15.892534 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 96362.5us
W20260503 13:58:15.857575 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 91471us
I20260503 13:58:15.893748 9491 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:15.900830 9642 tablet_bootstrap.cc:492] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:15.901526 9898 raft_consensus.cc:2804] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:15.933257 9898 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:15.933692 9898 raft_consensus.cc:3060] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:15.901325 9567 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:15.903622 9951 raft_consensus.cc:2749] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:15.897398 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.892485 9445 raft_consensus.cc:2468] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:15.907510 9644 tablet_bootstrap.cc:654] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
W20260503 13:58:15.940337 10043 log.cc:927] Time spent T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260503 13:58:15.910475 9858 consensus_queue.cc:1059] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260503 13:58:15.945442 9519 raft_consensus.cc:1240] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:15.945822 10034 raft_consensus.cc:3055] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:15.946231 10034 raft_consensus.cc:740] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:15.947461 10034 consensus_queue.cc:260] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.951035 9566 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:15.951535 9858 raft_consensus.cc:2804] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:15.952032 9858 raft_consensus.cc:697] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:15.952297 10016 raft_consensus.cc:2804] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:15.911746 9527 raft_consensus.cc:1240] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:15.912108 9522 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:15.912176 9950 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:15.952719 10016 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:15.953158 9950 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.953446 10016 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:15.955440 9950 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:15.916837 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.956300 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:15.894496 9416 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:15.957242 10034 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
W20260503 13:58:15.958122 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 63836.1us
I20260503 13:58:15.958834 9523 raft_consensus.cc:1240] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:15.959010 9416 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:15.950975 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 50045.9us
I20260503 13:58:15.961834 9567 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:15.963323 9567 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:15.964313 10017 raft_consensus.cc:2749] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:15.964915 10020 raft_consensus.cc:2749] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:15.964915 9995 raft_consensus.cc:2749] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:15.919229 9643 ts_tablet_manager.cc:1434] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.153s user 0.008s sys 0.001s
I20260503 13:58:15.965826 9995 raft_consensus.cc:2804] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:15.966192 9995 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:15.966570 9995 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:15.967062 9643 tablet_bootstrap.cc:492] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:15.919767 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:15.968957 9520 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.969897 9858 consensus_queue.cc:237] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.920316 9974 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.921784 10001 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:15.972783 10001 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:15.973205 10001 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:15.900403 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:15.809433 9978 consensus_queue.cc:237] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:15.925293 9991 consensus_queue.cc:1059] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260503 13:58:15.980247 9991 consensus_queue.cc:1059] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:15.925575 9987 consensus_queue.cc:1059] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260503 13:58:15.925498 9996 consensus_queue.cc:1059] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260503 13:58:15.926549 9491 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:15.992875 9416 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:16.015060 9643 tablet_bootstrap.cc:654] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:16.005856 9950 raft_consensus.cc:2749] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:15.995208 9898 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.017203 9525 raft_consensus.cc:1240] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:16.027285 9567 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:16.028403 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 102068us
I20260503 13:58:16.029253 9491 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:16.030691 9491 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:16.032006 9491 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:16.040596 9602 raft_consensus.cc:1240] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:16.041663 10060 raft_consensus.cc:3055] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Stepping down as leader of term 1
W20260503 13:58:16.042028 10050 log.cc:927] Time spent T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.214s user 0.001s sys 0.000s
I20260503 13:58:16.042253 9521 raft_consensus.cc:2468] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:16.043627 9526 raft_consensus.cc:1240] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:16.051726 9453 raft_consensus.cc:2468] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
W20260503 13:58:16.052827 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 113194us
I20260503 13:58:16.032406 9520 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:16.042055 10060 raft_consensus.cc:740] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:16.053130 9898 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:16.054364 9950 raft_consensus.cc:2749] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:16.066722 9724 raft_consensus.cc:2764] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260503 13:58:16.067304 10039 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:16.067728 10039 raft_consensus.cc:697] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:16.065402 9858 raft_consensus.cc:2749] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:16.069154 9974 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:16.082469 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 63662.3us
W20260503 13:58:16.083918 9567 leader_election.cc:343] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:16.068526 10039 consensus_queue.cc:237] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.071944 10057 raft_consensus.cc:3055] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:16.087741 10057 raft_consensus.cc:740] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:16.083253 10016 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.094024 10060 consensus_queue.cc:260] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.096639 9567 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:16.096917 10066 raft_consensus.cc:2804] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:16.097764 9491 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:16.099150 10066 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:16.107371 10066 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:16.107395 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:16.113426 9976 raft_consensus.cc:2804] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:16.113979 9976 raft_consensus.cc:697] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:16.114785 10001 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.114719 9567 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
W20260503 13:58:16.117476 9474 debug-util.cc:398] Leaking SignalData structure 0x7b08000a6c40 after lost signal to thread 8302
W20260503 13:58:16.118815 9474 debug-util.cc:398] Leaking SignalData structure 0x7b0800081920 after lost signal to thread 9385
W20260503 13:58:16.120131 9474 debug-util.cc:398] Leaking SignalData structure 0x7b08002280a0 after lost signal to thread 9477
W20260503 13:58:16.120391 9474 debug-util.cc:398] Leaking SignalData structure 0x7b08002280e0 after lost signal to thread 9478
I20260503 13:58:16.112528 9996 raft_consensus.cc:493] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:16.122299 9474 debug-util.cc:398] Leaking SignalData structure 0x7b0800210b80 after lost signal to thread 9552
W20260503 13:58:16.123471 9474 debug-util.cc:398] Leaking SignalData structure 0x7b0800296200 after lost signal to thread 9626
I20260503 13:58:16.126060 9996 raft_consensus.cc:515] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.128153 9996 leader_election.cc:290] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:16.110093 10037 consensus_queue.cc:1059] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260503 13:58:16.117147 10060 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
W20260503 13:58:16.131703 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 45656.5us
I20260503 13:58:16.114938 9976 consensus_queue.cc:237] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.134683 10016 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:16.135386 9418 leader_election.cc:343] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:16.088723 10057 consensus_queue.cc:260] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:16.132390 9975 raft_consensus.cc:2804] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:16.132341 9492 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:16.139009 9491 leader_election.cc:343] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:16.143124 9644 tablet_bootstrap.cc:492] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:16.138216 9975 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:16.145028 10014 raft_consensus.cc:2804] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:16.140830 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:16.144599 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 59491.6us
I20260503 13:58:16.152359 9642 tablet_bootstrap.cc:654] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
W20260503 13:58:16.145673 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 153006us
I20260503 13:58:16.155676 9416 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:16.157773 10068 consensus_queue.cc:1059] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260503 13:58:16.158324 10017 raft_consensus.cc:3055] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:16.158721 10017 raft_consensus.cc:740] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:16.159057 10068 raft_consensus.cc:3055] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:16.159253 9991 raft_consensus.cc:3055] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:16.159453 10068 raft_consensus.cc:740] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:16.159673 9991 raft_consensus.cc:740] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:16.162396 9951 raft_consensus.cc:2804] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:16.159546 10017 consensus_queue.cc:260] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.162861 9951 raft_consensus.cc:697] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:16.162688 9995 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.163419 10017 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:16.145820 9975 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:16.163792 9951 consensus_queue.cc:237] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.165781 9491 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:16.167169 9991 consensus_queue.cc:260] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:16.167870 9491 leader_election.cc:343] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:16.168399 9991 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:16.168900 9491 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:16.171908 10067 consensus_queue.cc:1048] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260503 13:58:16.172509 9526 raft_consensus.cc:1240] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:16.172752 9644 ts_tablet_manager.cc:1403] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.501s user 0.022s sys 0.032s
I20260503 13:58:16.151503 10014 raft_consensus.cc:697] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:16.171821 9950 raft_consensus.cc:2749] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:16.174036 10001 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:16.162225 10065 raft_consensus.cc:3055] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:16.177160 9643 tablet_bootstrap.cc:492] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:16.177655 9643 ts_tablet_manager.cc:1403] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.211s user 0.010s sys 0.008s
I20260503 13:58:16.179198 9453 raft_consensus.cc:1275] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:16.176905 10057 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:16.181664 9995 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:16.186661 9898 raft_consensus.cc:2749] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:16.160326 9996 consensus_queue.cc:1059] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
W20260503 13:58:16.189308 9491 leader_election.cc:343] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:16.193835 9978 consensus_queue.cc:1059] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260503 13:58:16.193960 10070 consensus_queue.cc:1059] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260503 13:58:16.188009 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:16.188265 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 51920.4us
I20260503 13:58:16.192807 9521 raft_consensus.cc:1275] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20260503 13:58:16.203397 9567 leader_election.cc:343] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:16.202488 10014 consensus_queue.cc:237] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.205258 10028 consensus_queue.cc:1048] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260503 13:58:16.187012 9995 raft_consensus.cc:2749] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:16.185014 10065 raft_consensus.cc:740] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
W20260503 13:58:16.202857 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 47329.5us
W20260503 13:58:16.193145 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:16.207868 9520 raft_consensus.cc:1275] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260503 13:58:16.210976 9416 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:16.213354 10001 raft_consensus.cc:3055] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:16.213454 9521 raft_consensus.cc:1275] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:16.213768 10001 raft_consensus.cc:740] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:16.214116 9416 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:16.214627 10001 consensus_queue.cc:260] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.215655 10067 raft_consensus.cc:2749] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:16.215863 10001 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:16.216182 10070 raft_consensus.cc:2804] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:16.216745 10070 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:16.217096 10070 raft_consensus.cc:3060] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:16.215389 9416 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:16.225440 10065 consensus_queue.cc:260] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.217294 9644 raft_consensus.cc:359] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.228276 9644 raft_consensus.cc:385] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:16.228469 9644 raft_consensus.cc:740] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:16.227593 9950 consensus_queue.cc:1059] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260503 13:58:16.229007 9644 consensus_queue.cc:260] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.229916 9950 raft_consensus.cc:3055] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:16.226702 10065 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:16.230260 9950 raft_consensus.cc:740] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:16.230906 9644 ts_tablet_manager.cc:1434] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.057s user 0.006s sys 0.000s
I20260503 13:58:16.226744 9452 raft_consensus.cc:1275] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260503 13:58:16.218637 10034 consensus_queue.cc:1048] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260503 13:58:16.218063 10068 consensus_queue.cc:260] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.237372 9528 raft_consensus.cc:1275] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:58:16.239099 9987 raft_consensus.cc:3055] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:16.238927 9950 consensus_queue.cc:260] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.239477 9987 raft_consensus.cc:740] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:16.240500 10067 raft_consensus.cc:2804] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:16.242517 10034 consensus_queue.cc:1048] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:16.243031 10067 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:16.243407 10067 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:16.243911 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 149) took 1004 ms. Trace:
I20260503 13:58:16.240321 9987 consensus_queue.cc:260] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.247900 9898 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:16.248406 9898 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.250490 9898 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:16.247339 9972 consensus_queue.cc:1048] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260503 13:58:16.234198 10039 consensus_queue.cc:1048] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260503 13:58:16.254729 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:16.255108 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:16.255473 9598 raft_consensus.cc:1275] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:16.255730 9596 raft_consensus.cc:3060] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:16.240345 9950 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
W20260503 13:58:16.257658 9416 leader_election.cc:343] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:16.259912 9416 leader_election.cc:304] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:16.260761 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:16.263190 9492 leader_election.cc:343] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:16.264943 10020 raft_consensus.cc:3055] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:16.265389 10020 raft_consensus.cc:740] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:16.248373 9987 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
W20260503 13:58:16.269866 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:16.271721 10068 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:16.240793 9418 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:16.304543 9598 raft_consensus.cc:1275] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:16.245966 9724 consensus_queue.cc:1059] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
W20260503 13:58:16.312422 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 54787.6us
I20260503 13:58:16.309376 9996 raft_consensus.cc:2749] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260503 13:58:16.319482 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 101254us
I20260503 13:58:16.319659 10073 raft_consensus.cc:2804] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:16.319844 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:16.320258 10073 raft_consensus.cc:697] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:16.321027 9995 consensus_queue.cc:1059] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:16.321225 10073 consensus_queue.cc:237] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.262810 9978 consensus_queue.cc:1048] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260503 13:58:16.330070 9416 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:16.320465 9453 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:16.266310 10020 consensus_queue.cc:260] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.348119 9644 tablet_bootstrap.cc:492] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:16.335920 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:16.352197 9976 consensus_queue.cc:1048] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260503 13:58:16.352944 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:16.353246 9643 raft_consensus.cc:359] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.353897 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:16.353916 9643 raft_consensus.cc:385] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:16.358098 9643 raft_consensus.cc:740] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:16.358714 9643 consensus_queue.cc:260] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.364116 9995 raft_consensus.cc:3055] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:16.364639 9995 raft_consensus.cc:740] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:16.356419 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:16.369445 9449 raft_consensus.cc:1275] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:16.377779 9351 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51388 (request call id 81) took 1578 ms. Trace:
I20260503 13:58:16.362953 9528 raft_consensus.cc:1275] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:16.246516 9597 rpcz_store.cc:276] 0503 13:58:15.239745 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:15.709889 (+470144us) service_pool.cc:224] Handling call
0503 13:58:15.710270 (+ 381us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:15.777705 (+ 67435us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:15.777735 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:15.778160 (+ 425us) log.cc:844] Serialized 26 byte log entry
0503 13:58:15.930741 (+152581us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b48001d01d8. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f46a631a1 00007f9f46a5a237 00007f9f46a5a0ca 00007f9f46b5192f 00007f9f46b515f7 00007f9f46b65c1c 00007f9f46b65bc8 00007f9f46b65b48 00007f9f46b65b03 00007f9f46b64dac 00007f9f45a8db36
0503 13:58:16.059577 (+128836us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:16.082149 (+ 22572us) log.cc:844] Serialized 14 byte log entry
0503 13:58:16.082377 (+ 228us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:16.082478 (+ 101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:16.092904 (+ 10426us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x7b18001f68a0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea8da33 00007f9f3ea8fe01 00007f9f3ea8fdb9 00007f9f3ea8fd49 00007f9f3ea8fd11 00007f9f3ea8f00d 0000000000379254 0000000000379099 00007f9f3ea70fba
0503 13:58:16.162189 (+ 69285us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b18001f68a0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea8db62 00007f9f3ea8fe01 00007f9f3ea8fdb9 00007f9f3ea8fd49 00007f9f3ea8fd11 00007f9f3ea8f00d 0000000000379254 0000000000379099 00007f9f3ea70fba
0503 13:58:16.242957 (+ 80768us) spinlock_profiling.cc:243] Waited 6.36 ms on lock 0x7b2400209b88. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e8dfe94 00007f9f3e8ddf9b 00007f9f3e96cd39 00007f9f3e96c9c5 00007f9f3e96c1fb 00007f9f416f2417 00007f9f416f2b09 00007f9f416f2d0a 00007f9f416f2edd 00007f9f416ca738
0503 13:58:16.243524 (+ 567us) raft_consensus.cc:1685] finished
0503 13:58:16.243546 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:16.243695 (+ 149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2307,"prepare.queue_time_us":2274,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":4204,"raft.run_cpu_time_us":3618,"raft.run_wall_time_us":150028,"spinlock_wait_cycles":28146560,"thread_start_us":238974,"threads_started":1,"wal-append.queue_time_us":277202}
I20260503 13:58:16.386612 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:16.342597 9996 consensus_queue.cc:1059] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260503 13:58:16.373649 10020 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:16.405620 9447 raft_consensus.cc:1275] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:16.417598 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:16.392510 10034 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:16.418684 9522 raft_consensus.cc:1240] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260503 13:58:16.419359 9449 raft_consensus.cc:1240] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:16.378152 9351 rpcz_store.cc:276] 0503 13:58:14.755060 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:14.755199 (+ 139us) service_pool.cc:224] Handling call
0503 13:58:16.325477 (+1570278us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0503 13:58:15.432102 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 13:58:15.471042 (+ 38940us) write_op.cc:432] Acquiring schema lock in shared mode
0503 13:58:15.471073 (+ 31us) write_op.cc:435] Acquired schema lock
0503 13:58:15.471092 (+ 19us) tablet.cc:662] Decoding operations
0503 13:58:15.471381 (+ 289us) write_op.cc:620] Acquiring the partition lock for write op
0503 13:58:15.471446 (+ 65us) write_op.cc:641] Partition lock acquired for write op
0503 13:58:15.471478 (+ 32us) tablet.cc:685] Acquiring locks for 1 operations
0503 13:58:15.471641 (+ 163us) tablet.cc:701] Row locks acquired
0503 13:58:15.471656 (+ 15us) write_op.cc:260] PREPARE: finished
0503 13:58:15.471774 (+ 118us) write_op.cc:270] Start()
0503 13:58:15.471871 (+ 97us) write_op.cc:276] Timestamp: P: 1777816695471740 usec, L: 0
0503 13:58:15.471891 (+ 20us) op_driver.cc:348] REPLICATION: starting
0503 13:58:15.550534 (+ 78643us) log.cc:844] Serialized 537 byte log entry
0503 13:58:16.117751 (+567217us) op_driver.cc:464] REPLICATION: finished
0503 13:58:16.186672 (+ 68921us) write_op.cc:301] APPLY: starting
0503 13:58:16.186768 (+ 96us) tablet.cc:1367] starting BulkCheckPresence
0503 13:58:16.186900 (+ 132us) tablet.cc:1370] finished BulkCheckPresence
0503 13:58:16.194766 (+ 7866us) tablet.cc:1372] starting ApplyRowOperation cycle
0503 13:58:16.202460 (+ 7694us) tablet.cc:1383] finished ApplyRowOperation cycle
0503 13:58:16.202661 (+ 201us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 13:58:16.202719 (+ 58us) write_op.cc:312] APPLY: finished
0503 13:58:16.203945 (+ 1226us) log.cc:844] Serialized 22 byte log entry
0503 13:58:16.204124 (+ 179us) write_op.cc:489] Releasing partition, row and schema locks
0503 13:58:16.204226 (+ 102us) write_op.cc:454] Released schema lock
0503 13:58:16.204270 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0503 13:58:16.220066 (+ 15796us) write_op.cc:489] Releasing partition, row and schema locks
0503 13:58:16.221301 (+ 1235us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":518059,"reported_tablets":3,"spinlock_wait_cycles":87680,"child_traces":[["op",{"apply.queue_time_us":52097,"apply.run_cpu_time_us":2809,"apply.run_wall_time_us":20019,"num_ops":1,"prepare.queue_time_us":60242,"prepare.run_cpu_time_us":10544,"prepare.run_wall_time_us":340338,"raft.queue_time_us":94553,"raft.run_cpu_time_us":12765,"raft.run_wall_time_us":25410,"replication_time_us":645578,"thread_start_us":230486,"threads_started":4,"wal-append.queue_time_us":215062}]]}
I20260503 13:58:16.425303 9524 raft_consensus.cc:1275] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:16.426277 10080 raft_consensus.cc:2749] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:16.377713 9898 consensus_queue.cc:1048] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260503 13:58:16.380146 9978 consensus_queue.cc:1048] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.054s
I20260503 13:58:16.365564 9995 consensus_queue.cc:260] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.433405 9642 tablet_bootstrap.cc:492] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:16.444962 9521 raft_consensus.cc:1275] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:58:16.449054 10016 consensus_queue.cc:1048] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:16.450160 9642 ts_tablet_manager.cc:1403] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.550s user 0.022s sys 0.055s
I20260503 13:58:16.451619 9643 ts_tablet_manager.cc:1434] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.274s user 0.005s sys 0.017s
I20260503 13:58:16.458280 9995 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:16.459260 9643 tablet_bootstrap.cc:492] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:16.445287 10034 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:16.737401 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 418652us
W20260503 13:58:16.745855 9566 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:16.749500 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:16.750272 9416 leader_election.cc:343] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:16.750656 9416 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:16.471390 9450 raft_consensus.cc:1275] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260503 13:58:16.751410 9450 pending_rounds.cc:85] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6: Aborting all ops after (but not including) 0
I20260503 13:58:16.751713 9450 pending_rounds.cc:107] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260503 13:58:16.752095 9450 raft_consensus.cc:2889] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:16.753635 9450 raft_consensus.cc:1275] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260503 13:58:16.649107 10078 log.cc:927] Time spent T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.124s user 0.000s sys 0.001s
W20260503 13:58:16.683338 9565 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:16.740475 9975 consensus_meta.cc:287] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.567s user 0.013s sys 0.005s
W20260503 13:58:16.686184 9991 consensus_meta.cc:287] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.517s user 0.009s sys 0.000s
I20260503 13:58:16.756458 9642 raft_consensus.cc:359] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.469355 9446 raft_consensus.cc:1240] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:16.675865 10079 raft_consensus.cc:3055] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:16.758888 9642 raft_consensus.cc:385] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:16.762517 9642 raft_consensus.cc:740] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:16.580880 10028 raft_consensus.cc:3055] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:16.763165 9642 consensus_queue.cc:260] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.552338 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:16.579563 9598 raft_consensus.cc:1275] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260503 13:58:16.768985 9644 tablet_bootstrap.cc:654] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
W20260503 13:58:16.548871 10074 log.cc:927] Time spent T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.222s user 0.001s sys 0.000s
I20260503 13:58:16.474215 9449 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:58:16.755136 9418 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:16.767491 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:16.468035 9951 consensus_queue.cc:1059] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.297s
I20260503 13:58:16.615247 9596 raft_consensus.cc:2468] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:16.500128 9864 consensus_queue.cc:1048] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.311s
I20260503 13:58:16.503851 10067 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.768378 9598 pending_rounds.cc:85] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868: Aborting all ops after (but not including) 1
I20260503 13:58:16.764690 10028 raft_consensus.cc:740] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:16.774469 9991 raft_consensus.cc:2749] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:16.567533 10070 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.593335 10066 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.768054 10079 raft_consensus.cc:740] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
W20260503 13:58:16.776126 9418 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:16.777599 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:16.786954 9448 raft_consensus.cc:1275] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260503 13:58:16.787312 9448 pending_rounds.cc:85] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6: Aborting all ops after (but not including) 0
I20260503 13:58:16.787549 9448 pending_rounds.cc:107] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260503 13:58:16.787789 9448 raft_consensus.cc:2889] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:16.780176 9598 pending_rounds.cc:107] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260503 13:58:16.791648 9642 ts_tablet_manager.cc:1434] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.339s user 0.018s sys 0.016s
I20260503 13:58:16.762830 9975 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.772756 9597 raft_consensus.cc:2468] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
W20260503 13:58:16.793812 10083 log.cc:927] Time spent T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.255s user 0.000s sys 0.001s
I20260503 13:58:16.793967 9445 raft_consensus.cc:2393] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:16.782814 9453 raft_consensus.cc:2468] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:16.796027 9444 raft_consensus.cc:1240] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:16.780696 9599 raft_consensus.cc:1275] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:16.804792 9602 raft_consensus.cc:2410] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260503 13:58:16.806802 9418 leader_election.cc:343] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:16.807179 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:16.807749 9520 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:16.808570 9416 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:16.802392 9491 leader_election.cc:343] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:16.797243 9598 raft_consensus.cc:2889] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:16.816862 9491 leader_election.cc:304] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:16.820298 9642 tablet_bootstrap.cc:492] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:16.803519 9450 raft_consensus.cc:1240] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:16.822813 10037 consensus_queue.cc:1048] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260503 13:58:16.846029 10067 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:16.849411 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 51447.9us
I20260503 13:58:16.825112 9991 raft_consensus.cc:2749] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260503 13:58:16.870918 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 99727.4us
I20260503 13:58:16.815052 10084 consensus_queue.cc:1059] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260503 13:58:16.851850 9525 raft_consensus.cc:1275] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:58:16.855654 9521 raft_consensus.cc:1240] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
W20260503 13:58:16.891053 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 104181us
I20260503 13:58:16.868003 9452 raft_consensus.cc:1275] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:16.847599 10028 consensus_queue.cc:260] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:16.937475 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 60937.5us
I20260503 13:58:16.875061 9447 raft_consensus.cc:1240] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
W20260503 13:58:16.947876 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 97797.1us
I20260503 13:58:17.035362 9491 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:16.877730 10079 consensus_queue.cc:260] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:16.903863 9527 raft_consensus.cc:1275] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:16.902958 10067 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:17.040705 10067 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.041534 10070 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.043531 9491 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:17.027915 9520 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:16.892663 10034 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:17.048444 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 240001us
W20260503 13:58:16.955190 10095 log.cc:927] Time spent T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260503 13:58:17.054533 9643 tablet_bootstrap.cc:654] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:16.899439 9974 raft_consensus.cc:2749] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:17.017866 9991 raft_consensus.cc:3055] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:17.009312 9522 raft_consensus.cc:1275] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:17.011394 9445 raft_consensus.cc:1275] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260503 13:58:16.905148 10085 log.cc:927] Time spent T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260503 13:58:16.918907 9975 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:16.971191 9973 raft_consensus.cc:3055] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:17.047894 10039 raft_consensus.cc:2749] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:16.919664 10066 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:16.900808 9951 consensus_queue.cc:1048] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.173s
I20260503 13:58:17.099188 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:17.099807 9526 raft_consensus.cc:2468] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:17.101318 9566 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:17.075886 9991 raft_consensus.cc:740] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:17.053639 10020 consensus_queue.cc:1048] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260503 13:58:17.095513 9524 raft_consensus.cc:1240] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260503 13:58:17.077623 10100 raft_consensus.cc:2749] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:17.093662 9973 raft_consensus.cc:740] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:17.103207 10070 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:17.107206 9972 raft_consensus.cc:2804] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:17.099295 10089 consensus_queue.cc:1048] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260503 13:58:17.102540 10034 consensus_queue.cc:1048] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260503 13:58:17.123147 9416 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:17.105209 10016 consensus_queue.cc:1048] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260503 13:58:17.105362 10014 consensus_queue.cc:1059] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260503 13:58:17.129201 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:17.129979 9526 raft_consensus.cc:2393] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:17.117115 10039 consensus_queue.cc:1048] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
W20260503 13:58:17.134236 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 66720us
I20260503 13:58:17.107719 9975 consensus_queue.cc:1059] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260503 13:58:17.124398 9520 raft_consensus.cc:1275] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:58:17.138705 10099 consensus_queue.cc:1048] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20260503 13:58:17.139531 9491 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:17.139833 10097 consensus_queue.cc:1048] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.093s
I20260503 13:58:17.131982 9973 consensus_queue.cc:260] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.145635 10014 raft_consensus.cc:3055] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:17.146046 10014 raft_consensus.cc:740] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:17.145673 9973 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:17.146545 9416 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:17.146889 10014 consensus_queue.cc:260] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.147972 10014 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:17.148424 10001 raft_consensus.cc:2764] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260503 13:58:17.130476 10066 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:17.145639 9523 raft_consensus.cc:1275] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:17.153026 10091 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:17.134811 10067 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.153508 10091 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.155540 10091 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:17.161087 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 58159.2us
I20260503 13:58:17.125658 9972 raft_consensus.cc:697] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:17.119879 10070 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.175968 9567 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:17.181200 10091 raft_consensus.cc:2804] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:17.184193 10091 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:17.186970 10070 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.187458 9642 tablet_bootstrap.cc:654] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:17.177381 10097 raft_consensus.cc:2804] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:17.190725 9491 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:17.131093 9991 consensus_queue.cc:260] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.196920 9972 consensus_queue.cc:237] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.205196 10108 raft_consensus.cc:2749] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:17.206451 9602 raft_consensus.cc:1275] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:17.207237 10073 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:17.207810 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:17.207711 10073 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.208889 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:17.209555 10073 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.210095 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:17.189119 10039 consensus_queue.cc:1048] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260503 13:58:17.210664 9520 raft_consensus.cc:2468] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:17.211874 9991 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
W20260503 13:58:17.214632 10096 log.cc:927] Time spent T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.115s user 0.001s sys 0.000s
I20260503 13:58:17.177675 10037 consensus_queue.cc:1048] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260503 13:58:17.177945 10034 consensus_queue.cc:1048] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260503 13:58:17.179123 10089 consensus_queue.cc:1048] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260503 13:58:17.189698 10097 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:17.218190 10097 raft_consensus.cc:3060] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:17.218837 10079 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:17.219235 9597 raft_consensus.cc:1275] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:17.219763 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:17.220197 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:17.220753 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:17.222146 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:17.223037 9595 raft_consensus.cc:2410] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:17.224233 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:17.224713 10016 consensus_queue.cc:1048] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260503 13:58:17.247973 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:17.189844 10091 raft_consensus.cc:3060] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:17.253587 9643 tablet_bootstrap.cc:492] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:17.257534 9643 ts_tablet_manager.cc:1403] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.800s user 0.018s sys 0.021s
I20260503 13:58:17.252642 10084 consensus_queue.cc:1059] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260503 13:58:17.267119 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:17.267374 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:17.268026 9600 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:17.268435 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:17.269654 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 154651us
I20260503 13:58:17.271370 9449 raft_consensus.cc:1275] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:17.272338 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:17.273514 9601 raft_consensus.cc:1275] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:17.275128 9724 consensus_queue.cc:1048] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:17.265461 9418 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:17.250656 9974 raft_consensus.cc:493] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:17.300825 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:17.300999 10028 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:17.302398 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:17.300305 9724 consensus_queue.cc:1048] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
W20260503 13:58:17.308007 9491 consensus_peers.cc:597] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a -> Peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Couldn't send request to peer 4bea375a55034750a1f8804b0db97868. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260503 13:58:17.308501 10066 raft_consensus.cc:2749] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:17.310783 9600 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:17.253305 10039 consensus_queue.cc:1048] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260503 13:58:17.315001 9951 raft_consensus.cc:3055] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:17.315346 9951 raft_consensus.cc:740] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:17.277034 10001 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:17.318363 10001 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.320199 10001 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.240724 9898 consensus_queue.cc:1048] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.099s
I20260503 13:58:17.320330 10097 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.322904 9974 raft_consensus.cc:515] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:17.330662 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 153683us
W20260503 13:58:17.338490 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 160210us
I20260503 13:58:17.316100 9951 consensus_queue.cc:260] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.331084 10073 raft_consensus.cc:3055] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:17.351032 9491 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:17.338119 10016 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:17.360726 9644 tablet_bootstrap.cc:492] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:17.364393 9451 raft_consensus.cc:1275] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:17.365520 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:17.368475 9644 ts_tablet_manager.cc:1403] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 1.027s user 0.029s sys 0.028s
I20260503 13:58:17.371374 10089 consensus_queue.cc:1048] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260503 13:58:17.372227 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:17.373219 10089 consensus_queue.cc:1048] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:17.374394 10089 consensus_queue.cc:1059] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:17.377732 10110 raft_consensus.cc:2804] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:17.396607 9978 consensus_queue.cc:1048] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260503 13:58:17.398686 10091 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:17.221405 10067 consensus_queue.cc:1175] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4bea375a55034750a1f8804b0db97868" responder_term: 3 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260503 13:58:17.350535 10073 raft_consensus.cc:740] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:17.347743 10097 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.357976 10001 raft_consensus.cc:2804] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election won for term 3
W20260503 13:58:17.406623 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 108372us
I20260503 13:58:17.407537 9416 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:17.357020 9991 consensus_queue.cc:1048] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260503 13:58:17.399791 10070 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:17.418476 9416 leader_election.cc:343] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:17.424039 9492 leader_election.cc:343] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:17.424144 9416 leader_election.cc:304] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:17.416944 10066 raft_consensus.cc:2749] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:17.417192 10028 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:17.401913 10110 raft_consensus.cc:697] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:17.421288 9593 raft_consensus.cc:2393] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 1: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:17.428820 10028 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.426573 10001 raft_consensus.cc:697] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:17.432945 10028 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.433359 10001 consensus_queue.cc:237] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.433647 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:17.433568 10073 consensus_queue.cc:260] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.434279 9448 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:17.435510 9492 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:17.394299 10016 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.430047 9446 raft_consensus.cc:1275] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:58:17.423039 10089 raft_consensus.cc:3055] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:17.452462 10110 consensus_queue.cc:237] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.400631 9864 consensus_queue.cc:1048] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20260503 13:58:17.458415 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:17.459038 9598 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:17.455950 10066 raft_consensus.cc:2749] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:17.460157 9491 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:17.461210 10039 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:17.461652 10039 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:17.460834 10016 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:17.402163 10099 consensus_queue.cc:1048] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
W20260503 13:58:17.429872 9416 leader_election.cc:343] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:17.426854 10070 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.460827 9898 raft_consensus.cc:2749] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260503 13:58:17.451668 10028 raft_consensus.cc:2804] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:17.484652 10028 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:17.485021 10070 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.485044 10028 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:17.489898 9599 raft_consensus.cc:2410] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:17.491618 9448 raft_consensus.cc:1275] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:58:17.451151 10089 raft_consensus.cc:740] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:17.495640 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:17.498518 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:17.499110 9444 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:17.492627 10124 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:17.502718 10066 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:17.509047 10124 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.503297 10089 consensus_queue.cc:260] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.510058 10089 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:17.511010 10124 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.503638 9446 raft_consensus.cc:1275] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260503 13:58:17.523300 9446 pending_rounds.cc:85] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6: Aborting all ops after (but not including) 1
I20260503 13:58:17.524662 9446 pending_rounds.cc:107] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260503 13:58:17.524906 9446 raft_consensus.cc:2889] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260503 13:58:17.447364 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:17.509111 10066 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:17.500092 9974 leader_election.cc:290] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:17.507087 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 252025us
I20260503 13:58:17.536690 9416 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:17.535033 9418 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:17.555629 9643 raft_consensus.cc:359] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.566184 9643 raft_consensus.cc:385] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:17.571345 9643 raft_consensus.cc:740] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:17.513296 10091 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:17.574249 9643 consensus_queue.cc:260] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.575840 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:17.525578 9448 raft_consensus.cc:1275] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:58:17.525985 10073 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:17.581765 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:17.563176 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:17.594184 9445 raft_consensus.cc:2468] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:17.595125 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:17.595693 9445 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:17.574298 10070 raft_consensus.cc:2749] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:17.569703 9974 consensus_queue.cc:1048] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260503 13:58:17.573694 10039 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:17.601414 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:17.602176 9593 raft_consensus.cc:2393] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:17.592294 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:17.604895 10066 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.605260 9599 raft_consensus.cc:2393] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 2: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:17.605908 10028 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.607846 10028 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.574499 10014 consensus_queue.cc:1048] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260503 13:58:17.586930 10037 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:17.610075 10037 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.611830 10037 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:17.612619 10112 log.cc:927] Time spent T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
I20260503 13:58:17.620927 9492 leader_election.cc:304] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:17.622046 9492 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:17.622740 10039 raft_consensus.cc:2804] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:17.623086 10039 raft_consensus.cc:493] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:17.623405 10039 raft_consensus.cc:3060] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:17.563038 9951 raft_consensus.cc:3060] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:17.641098 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:17.637457 10028 raft_consensus.cc:2804] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260503 13:58:17.652060 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 117243us
I20260503 13:58:17.652796 9418 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:17.654845 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:17.656308 9566 leader_election.cc:343] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:17.657460 9566 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:17.663969 9643 ts_tablet_manager.cc:1434] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.405s user 0.005s sys 0.004s
I20260503 13:58:17.664844 10014 consensus_queue.cc:1048] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:17.653208 10107 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:17.655504 9521 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:17.652680 10028 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:17.658656 10067 raft_consensus.cc:2749] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:17.667339 10066 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:17.677549 9418 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:17.679116 10124 raft_consensus.cc:2749] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:17.664505 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:17.669687 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:17.660892 10016 consensus_queue.cc:1048] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260503 13:58:17.674984 10037 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:17.696312 9418 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:17.676569 10028 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:17.660797 10113 consensus_queue.cc:1048] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260503 13:58:17.669090 9864 consensus_queue.cc:1048] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260503 13:58:17.707003 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:17.709092 9644 raft_consensus.cc:359] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.713501 9644 raft_consensus.cc:385] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:17.713794 9644 raft_consensus.cc:740] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:17.714406 9644 consensus_queue.cc:260] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.719281 9642 tablet_bootstrap.cc:492] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:17.719755 9642 ts_tablet_manager.cc:1403] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.900s user 0.006s sys 0.026s
I20260503 13:58:17.720455 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:17.680027 10066 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.673461 10107 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
W20260503 13:58:17.721859 9566 leader_election.cc:343] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:17.732740 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:17.733640 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:17.734081 10129 raft_consensus.cc:2804] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:17.734220 9599 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:17.734530 10129 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:17.734891 10129 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
W20260503 13:58:17.731210 9491 leader_election.cc:343] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:17.724679 9524 raft_consensus.cc:2393] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:17.746307 10091 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:17.746846 10091 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.723484 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:17.758200 9642 raft_consensus.cc:359] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.758909 9642 raft_consensus.cc:385] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:17.759183 9642 raft_consensus.cc:740] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:17.759796 9642 consensus_queue.cc:260] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.768061 10091 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:17.726976 10097 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:17.725205 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:17.787901 9593 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:17.797236 10097 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.785188 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:17.782424 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:17.816812 9451 raft_consensus.cc:2393] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:17.828488 9644 ts_tablet_manager.cc:1434] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.460s user 0.009s sys 0.000s
I20260503 13:58:17.838027 9521 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:17.785987 10124 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:17.839004 10124 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.845029 9642 ts_tablet_manager.cc:1434] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.125s user 0.004s sys 0.003s
I20260503 13:58:17.846596 10066 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.822971 9447 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 3
I20260503 13:58:17.883862 9644 tablet_bootstrap.cc:492] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:17.881918 9864 raft_consensus.cc:493] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:17.885361 9864 raft_consensus.cc:515] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:17.888900 9864 leader_election.cc:290] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:17.889753 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:17.894197 10107 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:17.907600 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:17.908828 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:17.909003 9565 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:17.909936 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:17.911024 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:17.911221 9418 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:17.911667 9523 raft_consensus.cc:2368] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3.
W20260503 13:58:17.898880 10137 log.cc:927] Time spent T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260503 13:58:17.891269 10124 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.912456 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:17.984274 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:17.984894 9450 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
W20260503 13:58:17.972916 10135 log.cc:927] Time spent T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.095s user 0.000s sys 0.001s
I20260503 13:58:17.985410 9416 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:17.974480 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:17.986418 9416 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:17.986871 9450 raft_consensus.cc:1240] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260503 13:58:17.987174 10066 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:17.987632 9522 raft_consensus.cc:1240] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
W20260503 13:58:17.988844 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 77692.5us
I20260503 13:58:17.978199 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:17.956375 9974 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:17.988426 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:17.987602 10066 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:17.960825 10130 log.cc:927] Time spent T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260503 13:58:17.904651 10039 raft_consensus.cc:515] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.003696 9418 leader_election.cc:400] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805) with higher term. Message: Invalid argument: T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3.
I20260503 13:58:18.003844 9492 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:18.004070 9418 leader_election.cc:403] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260503 13:58:17.988711 10067 raft_consensus.cc:2749] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260503 13:58:18.068629 10132 log.cc:927] Time spent T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
I20260503 13:58:18.052093 10125 raft_consensus.cc:2804] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:18.069958 10125 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:18.070345 10125 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:18.052598 10128 raft_consensus.cc:2749] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:18.054157 10079 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:18.054708 10026 raft_consensus.cc:2804] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:18.059705 10097 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.997723 9452 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:18.060825 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:18.065272 10066 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.995034 9974 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.104890 9643 tablet_bootstrap.cc:492] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:18.106202 9974 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:17.996793 9593 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:18.067062 10070 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:18.107089 10070 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.095898 10026 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:18.109244 10026 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:18.104058 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:18.096130 10079 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.097414 9599 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:18.814318 9416 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:18.819118 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 246) took 1173 ms. Trace:
W20260503 13:58:18.814035 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 824248us
I20260503 13:58:18.821312 9418 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:18.274360 9567 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.041s sys 0.070s
W20260503 13:58:18.836865 9567 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.134s user 0.048s sys 0.107s
W20260503 13:58:18.837159 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 1.13416e+06us
W20260503 13:58:18.292842 10139 log.cc:927] Time spent T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
W20260503 13:58:18.308542 9951 consensus_meta.cc:287] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.666s user 0.005s sys 0.005s
W20260503 13:58:18.590704 10028 consensus_meta.cc:287] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.879s user 0.007s sys 0.000s
W20260503 13:58:18.596098 10129 consensus_meta.cc:287] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.861s user 0.005s sys 0.004s
W20260503 13:58:18.594410 10125 consensus_meta.cc:287] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.524s user 0.002s sys 0.005s
W20260503 13:58:18.592685 10143 log.cc:927] Time spent T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.464s user 0.002s sys 0.000s
I20260503 13:58:18.153504 9974 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:18.809193 9447 consensus_meta.cc:287] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.921s user 0.006s sys 0.003s
W20260503 13:58:18.587525 10144 log.cc:927] Time spent T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.331s user 0.000s sys 0.001s
W20260503 13:58:18.595486 10156 log.cc:927] Time spent T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.421s user 0.001s sys 0.001s
W20260503 13:58:18.132267 10034 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.028s sys 0.059s
W20260503 13:58:18.265314 10017 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.025s sys 0.084s
W20260503 13:58:18.450014 9565 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:18.228933 10134 log.cc:927] Time spent T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20260503 13:58:18.837787 9567 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
W20260503 13:58:18.456301 10152 log.cc:927] Time spent T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.270s user 0.001s sys 0.000s
I20260503 13:58:18.819777 9526 rpcz_store.cc:276] 0503 13:58:17.639136 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.643098 (+ 3962us) service_pool.cc:224] Handling call
0503 13:58:17.646345 (+ 3247us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:17.646664 (+ 319us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:17.646694 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:17.669655 (+ 22961us) spinlock_profiling.cc:243] Waited 3.26 ms on lock 0x7b2c000655f8. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f40425c81 00007f9f404355e2 00007f9f41b66de1 00007f9f41b2b81a 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22
0503 13:58:17.674384 (+ 4729us) log.cc:844] Serialized 26 byte log entry
0503 13:58:17.726802 (+ 52418us) spinlock_profiling.cc:243] Waited 2.26 ms on lock 0x7b48000fc0d8. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f46a631a1 00007f9f46a5a237 00007f9f46a5a0ca 00007f9f46b5192f 00007f9f46b515f7 00007f9f46b65c1c 00007f9f46b65bc8 00007f9f46b65b48 00007f9f46b65b03 00007f9f46b64dac 00007f9f45a8db36
0503 13:58:18.173341 (+446539us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:18.173380 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:18.173443 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:18.312981 (+139538us) raft_consensus.cc:1685] finished
0503 13:58:18.313003 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:18.812780 (+499777us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1306,"raft.run_cpu_time_us":1198,"raft.run_wall_time_us":87389,"spinlock_wait_cycles":12691456,"thread_start_us":498237,"threads_started":1,"wal-append.queue_time_us":637088}
I20260503 13:58:18.470512 9594 raft_consensus.cc:1275] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260503 13:58:18.261909 10092 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:18.262530 9453 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.025s sys 0.045s
I20260503 13:58:18.450678 10070 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:18.871493 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 257) took 1210 ms. Trace:
W20260503 13:58:18.866712 10034 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.279s user 0.046s sys 0.102s
I20260503 13:58:18.555572 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:18.439301 9449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 210) took 1155 ms. Trace:
I20260503 13:58:18.194283 10037 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:18.554842 9991 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:18.293987 9522 raft_consensus.cc:1275] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:18.384809 10100 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:18.265724 10039 leader_election.cc:290] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:18.825709 10073 raft_consensus.cc:2804] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:18.833436 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 186) took 1096 ms. Trace:
I20260503 13:58:18.580055 10026 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.831483 10097 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:18.863018 9973 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
W20260503 13:58:18.868573 10017 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.187s user 0.052s sys 0.103s
I20260503 13:58:18.862881 10110 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:18.866998 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 161) took 1293 ms. Trace:
I20260503 13:58:18.863319 9447 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:18.865685 9974 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:18.891657 9898 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.017s sys 0.020s
W20260503 13:58:18.876221 9453 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.209s user 0.039s sys 0.079s
I20260503 13:58:18.863953 10129 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.864017 10028 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:18.874464 9594 pending_rounds.cc:85] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868: Aborting all ops after (but not including) 1
I20260503 13:58:18.892933 9643 tablet_bootstrap.cc:654] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:18.902313 9522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 179) took 1184 ms. Trace:
I20260503 13:58:18.876052 10070 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:18.872885 9951 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:18.863808 10125 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:18.892266 10037 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:18.889503 10091 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
W20260503 13:58:18.929296 10147 log.cc:927] Time spent T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260503 13:58:18.929147 10091 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.892714 10073 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:18.897498 10160 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:18.931569 9520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 249) took 1287 ms. Trace:
I20260503 13:58:18.931941 10091 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:18.931676 10160 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.932247 9444 raft_consensus.cc:2393] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
W20260503 13:58:18.909080 9447 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 220) took 1276 ms (client timeout 1628 ms). Trace:
I20260503 13:58:18.894707 10097 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.901974 9973 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.912914 10039 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:18.933246 10166 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:18.875965 9528 rpcz_store.cc:276] 0503 13:58:17.657690 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.660312 (+ 2622us) service_pool.cc:224] Handling call
0503 13:58:18.092251 (+431939us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b5800004598. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50a3e 000000000035a3c9 00007f9f46a888e0 00007f9f46b51784 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:18.092421 (+ 170us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:18.825680 (+733259us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:18.835182 (+ 9502us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:18.854268 (+ 19086us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:18.856019 (+ 1751us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:18.861118 (+ 5099us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:18.868391 (+ 7273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25864960}
I20260503 13:58:18.933764 10166 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.934099 10039 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.913527 9974 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:18.914140 9524 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:18.914980 10028 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:18.901136 10122 raft_consensus.cc:2749] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805) with higher term. Message: Invalid argument: T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3.
W20260503 13:58:18.902797 9898 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s user 0.017s sys 0.020s
I20260503 13:58:18.904423 10034 raft_consensus.cc:2804] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:18.893450 10100 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.891920 9599 rpcz_store.cc:276] 0503 13:58:17.727719 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.733388 (+ 5669us) service_pool.cc:224] Handling call
0503 13:58:18.823907 (+1090519us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:18.918987 9451 raft_consensus.cc:1240] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260503 13:58:18.900964 10159 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:18.933113 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 232) took 1434 ms. Trace:
I20260503 13:58:18.895905 9991 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.900084 10110 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.910007 9594 pending_rounds.cc:107] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260503 13:58:18.931072 10125 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:18.931198 10073 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:18.903318 10151 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:18.917608 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:18.905237 9596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 178) took 1553 ms. Trace:
I20260503 13:58:18.934121 10097 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:18.985482 10167 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:18.985644 10097 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:18.985932 10099 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:18.986125 10097 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.986387 10099 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.935412 9600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 177) took 1580 ms. Trace:
I20260503 13:58:18.990147 10099 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:18.990360 9600 rpcz_store.cc:276] 0503 13:58:17.350279 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.356839 (+ 6560us) service_pool.cc:224] Handling call
0503 13:58:17.369357 (+ 12518us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:17.398800 (+ 29443us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:17.398836 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:17.399291 (+ 455us) log.cc:844] Serialized 26 byte log entry
0503 13:58:17.659904 (+260613us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:17.662237 (+ 2333us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:17.670553 (+ 8316us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:18.916191 (+1245638us) raft_consensus.cc:1685] finished
0503 13:58:18.916212 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:18.930375 (+ 14163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25481,"raft.queue_time_us":29320,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":249681,"threads_started":1,"wal-append.queue_time_us":519879}
I20260503 13:58:18.899030 10092 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.993511 10092 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:18.993901 10067 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:18.919317 10070 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.968252 10026 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:18.966449 10034 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:18.975958 9594 raft_consensus.cc:2889] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:18.911509 9522 rpcz_store.cc:276] 0503 13:58:17.720366 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:18.293299 (+572933us) service_pool.cc:224] Handling call
0503 13:58:18.293648 (+ 349us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:18.892810 (+599162us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:18.900946 (+ 8136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:18.982897 10151 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.984414 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:18.978758 9566 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:18.978324 10017 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:18.989423 9523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 262) took 1298 ms. Trace:
I20260503 13:58:18.989907 10097 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:18.972124 9974 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:18.982406 9644 tablet_bootstrap.cc:654] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:18.994522 10067 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.978902 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 247) took 1333 ms. Trace:
I20260503 13:58:18.999182 9642 tablet_bootstrap.cc:492] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:18.996542 10167 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.988161 10166 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:18.932586 9447 rpcz_store.cc:269] 0503 13:58:17.625826 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.684169 (+ 58343us) service_pool.cc:224] Handling call
0503 13:58:18.902158 (+1217989us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:18.993947 10092 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:18.896950 9598 rpcz_store.cc:276] 0503 13:58:17.567379 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.577648 (+ 10269us) service_pool.cc:224] Handling call
0503 13:58:17.593440 (+ 15792us) spinlock_profiling.cc:243] Waited 969 us on lock 0x7b5800010898. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:17.593609 (+ 169us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:17.647010 (+ 53401us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:17.647046 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:17.647510 (+ 464us) log.cc:844] Serialized 26 byte log entry
0503 13:58:18.051950 (+404440us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:18.052002 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:18.052081 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:18.317761 (+265680us) raft_consensus.cc:1685] finished
0503 13:58:18.323153 (+ 5392us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:18.860793 (+537640us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61891,"raft.queue_time_us":50371,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"spinlock_wait_cycles":2228480,"thread_start_us":385000,"threads_started":1,"wal-append.queue_time_us":662300}
I20260503 13:58:19.001348 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:18.985387 10079 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:18.927404 9951 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:18.977772 10173 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:18.994910 10125 raft_consensus.cc:2749] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:19.001932 10034 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
W20260503 13:58:19.022941 10164 log.cc:927] Time spent T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260503 13:58:19.016703 9525 rpcz_store.cc:276] 0503 13:58:17.643617 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.644827 (+ 1210us) service_pool.cc:224] Handling call
0503 13:58:17.670113 (+ 25286us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:17.674033 (+ 3920us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:17.674067 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:17.674468 (+ 401us) log.cc:844] Serialized 26 byte log entry
0503 13:58:17.958262 (+283794us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:17.958312 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:17.960145 (+ 1833us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:18.455766 (+495621us) spinlock_profiling.cc:243] Waited 8.28 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:18.935417 (+479651us) raft_consensus.cc:1685] finished
0503 13:58:18.956315 (+ 20898us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:18.977071 (+ 20756us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":458,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"spinlock_wait_cycles":19041152,"thread_start_us":283436,"threads_started":1,"wal-append.queue_time_us":503511}
W20260503 13:58:19.004976 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 183746us
I20260503 13:58:19.026309 9492 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:19.028789 10001 raft_consensus.cc:2749] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260503 13:58:19.032441 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 161962us
I20260503 13:58:19.018273 10039 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:18.977214 10159 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.019778 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:19.009475 10017 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.030040 10129 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:19.039083 9567 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:18.984884 9596 rpcz_store.cc:276] 0503 13:58:17.350571 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.367300 (+ 16729us) service_pool.cc:224] Handling call
0503 13:58:17.426497 (+ 59197us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:17.452077 (+ 25580us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:17.452112 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:17.562624 (+110512us) log.cc:844] Serialized 26 byte log entry
0503 13:58:17.921881 (+359257us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:17.956595 (+ 34714us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:17.960103 (+ 3508us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:18.896867 (+936764us) raft_consensus.cc:1685] finished
0503 13:58:18.896888 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:18.903761 (+ 6873us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10535,"raft.queue_time_us":21459,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"thread_start_us":342661,"threads_started":1,"wal-append.queue_time_us":590949}
I20260503 13:58:19.029462 10166 raft_consensus.cc:2804] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:19.034472 9444 raft_consensus.cc:2468] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:18.931960 9520 rpcz_store.cc:276] 0503 13:58:17.644639 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.647034 (+ 2395us) service_pool.cc:224] Handling call
0503 13:58:17.647346 (+ 312us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:17.647668 (+ 322us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:17.647698 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:17.885516 (+237818us) spinlock_profiling.cc:243] Waited 4.23 ms on lock 0x7b1800299ce0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea5bbe6 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f40425c81 00007f9f404355e2 00007f9f41b66de1 00007f9f41b2b81a
0503 13:58:17.885898 (+ 382us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b2c000655f8. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f40425c81 00007f9f404355e2 00007f9f41b66de1 00007f9f41b2b81a 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22
0503 13:58:17.886354 (+ 456us) log.cc:844] Serialized 26 byte log entry
0503 13:58:18.321718 (+435364us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:18.321760 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:18.321843 (+ 83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:18.930034 (+608191us) raft_consensus.cc:1685] finished
0503 13:58:18.931380 (+ 1346us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:18.931513 (+ 133us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":369,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":7184,"spinlock_wait_cycles":21352576,"thread_start_us":403430,"threads_started":1,"wal-append.queue_time_us":992214}
I20260503 13:58:19.044389 10167 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:19.052868 9643 tablet_bootstrap.cc:492] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:19.079720 9643 ts_tablet_manager.cc:1403] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.975s user 0.023s sys 0.023s
I20260503 13:58:19.029889 10079 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:19.036600 9418 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:19.059249 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 53851.8us
I20260503 13:58:19.018469 9974 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.074354 10100 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:19.080335 10170 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:19.025240 10092 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.061443 10039 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:19.063251 10125 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:19.054973 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:19.055106 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:19.081190 10079 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.032254 10173 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.111368 9492 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:19.574874 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:19.632942 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 521644us
I20260503 13:58:19.531847 9599 raft_consensus.cc:1275] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:19.635588 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:19.636909 9598 raft_consensus.cc:2393] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:19.636821 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:19.637416 9599 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:19.638273 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:19.638545 9447 raft_consensus.cc:2393] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 1: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:19.639063 9599 raft_consensus.cc:2410] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:19.639710 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:19.639780 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:19.640345 9447 raft_consensus.cc:3060] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:19.157793 9991 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:19.641193 9991 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.641692 9991 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.541325 10179 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:19.266657 9450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 244) took 1652 ms. Trace:
I20260503 13:58:19.045039 10168 raft_consensus.cc:2804] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:19.644083 10184 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:18.974011 9453 rpcz_store.cc:276] 0503 13:58:17.498739 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.498884 (+ 145us) service_pool.cc:224] Handling call
0503 13:58:17.586775 (+ 87891us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:17.664828 (+ 78053us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:17.664856 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:17.665469 (+ 613us) log.cc:844] Serialized 26 byte log entry
0503 13:58:18.912477 (+1247008us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:18.915539 (+ 3062us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:18.927369 (+ 11830us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:18.930082 (+ 2713us) raft_consensus.cc:1685] finished
0503 13:58:18.932896 (+ 2814us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:18.933054 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80020,"raft.queue_time_us":74066,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":12785,"thread_start_us":1238523,"threads_started":1,"wal-append.queue_time_us":1147292}
W20260503 13:58:19.644500 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 561593us
I20260503 13:58:19.168660 10026 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.129554 10169 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.088970 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 218) took 1552 ms. Trace:
I20260503 13:58:19.647065 10039 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.094895 10170 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.204466 10073 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.075923 10166 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260503 13:58:19.650918 10034 consensus_meta.cc:287] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.612s user 0.004s sys 0.017s
I20260503 13:58:19.650892 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 185) took 1944 ms. Trace:
I20260503 13:58:19.094149 10070 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:19.631328 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 592266us
I20260503 13:58:19.209331 10067 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:19.083006 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:19.133496 10089 raft_consensus.cc:2749] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260503 13:58:19.631335 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 594786us
I20260503 13:58:19.637715 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:19.655575 9644 tablet_bootstrap.cc:492] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:19.733916 9527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 248) took 2088 ms. Trace:
I20260503 13:58:19.657487 10159 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:19.659356 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 235) took 2159 ms. Trace:
I20260503 13:58:19.660408 10079 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:19.096086 10129 raft_consensus.cc:493] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:19.661186 9447 raft_consensus.cc:2468] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:19.663034 10191 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:19.006435 9523 rpcz_store.cc:276] 0503 13:58:17.691191 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.984432 (+293241us) service_pool.cc:224] Handling call
0503 13:58:17.985004 (+ 572us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:18.150192 (+165188us) raft_consensus.cc:1534] Early marking committed up to index 1
0503 13:58:18.204458 (+ 54266us) log.cc:844] Serialized 14 byte log entry
0503 13:58:18.442980 (+238522us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b1800086560. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea8db97 00007f9f3ea8fe01 00007f9f3ea8fdb9 00007f9f3ea8fd49 00007f9f3ea8fd11 00007f9f3ea8f00d 0000000000379254 0000000000379099 00007f9f3ea70fba
0503 13:58:18.876816 (+433836us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:18.885572 (+ 8756us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:18.885626 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:18.989186 (+103560us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540027c4d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:18.989237 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:18.989370 (+ 133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18899,"prepare.queue_time_us":28648,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":7216,"raft.queue_time_us":177848,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":3668,"spinlock_wait_cycles":295706240,"thread_start_us":386806,"threads_started":2,"wal-append.queue_time_us":627192}
I20260503 13:58:18.890682 9449 rpcz_store.cc:276] 0503 13:58:17.275718 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.298898 (+ 23180us) service_pool.cc:224] Handling call
0503 13:58:17.301326 (+ 2428us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:17.505974 (+204648us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:17.506015 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:17.506470 (+ 455us) log.cc:844] Serialized 26 byte log entry
0503 13:58:17.502421 (+ -4049us) spinlock_profiling.cc:243] Waited 25.6 ms on lock 0x7b48000f0558. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f46a631a1 00007f9f46a5a237 00007f9f46a5a0ca 00007f9f46b5192f 00007f9f46b515f7 00007f9f46b65c1c 00007f9f46b65bc8 00007f9f46b65b48 00007f9f46b65b03 00007f9f46b64dac 00007f9f45a8db36
0503 13:58:17.880691 (+378270us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:17.906043 (+ 25352us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:17.921983 (+ 15940us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:18.368864 (+446881us) raft_consensus.cc:1685] finished
0503 13:58:18.368888 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:18.400018 (+ 31130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":131137,"raft.queue_time_us":57476,"raft.run_cpu_time_us":1305,"raft.run_wall_time_us":163543,"spinlock_wait_cycles":59002112,"thread_start_us":470496,"threads_started":2,"wal-append.queue_time_us":452758}
W20260503 13:58:19.667047 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:19.763774 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:19.730672 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:19.730515 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 461665us
I20260503 13:58:19.209043 10122 raft_consensus.cc:493] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:19.267283 10017 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:19.209811 10160 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:19.753192 9598 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:19.208873 10175 raft_consensus.cc:2749] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:19.739104 9643 raft_consensus.cc:359] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.186846 10158 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.152283 10167 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:19.642732 10179 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.657441 9951 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:19.111251 10155 log.cc:927] Time spent T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.911s user 0.002s sys 0.002s
I20260503 13:58:19.195113 10171 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.207614 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:19.208638 10092 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:19.201995 10151 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:19.151793 10100 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.152431 9898 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.154518 10163 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.740130 10166 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:19.741621 9991 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:19.747100 9644 ts_tablet_manager.cc:1403] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 1.871s user 0.009s sys 0.034s
I20260503 13:58:19.075948 10097 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:19.732086 10168 raft_consensus.cc:697] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:19.741828 9596 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:19.727201 10125 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.206779 10037 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.758380 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:19.763773 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:19.209208 9950 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:19.734037 10026 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.745790 10070 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.768851 9567 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:19.750095 9602 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:19.792714 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:19.741786 10169 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.778823 9643 raft_consensus.cc:385] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:19.742058 10034 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.749904 10067 consensus_queue.cc:1059] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260503 13:58:19.802193 9950 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.781314 9453 raft_consensus.cc:2368] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3.
I20260503 13:58:19.787263 9491 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:19.765928 10079 consensus_queue.cc:1048] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260503 13:58:19.776187 10129 raft_consensus.cc:515] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.782333 10186 raft_consensus.cc:2804] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:19.738368 10184 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.788941 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:19.793417 10185 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:19.797060 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:19.790429 10194 consensus_queue.cc:1048] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260503 13:58:19.766816 10073 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:19.791399 10188 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:19.792531 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:19.805307 9643 raft_consensus.cc:740] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:19.804292 10190 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:19.749478 9527 rpcz_store.cc:276] 0503 13:58:17.644044 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.646046 (+ 2002us) service_pool.cc:224] Handling call
0503 13:58:17.646413 (+ 367us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:17.646776 (+ 363us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:17.646809 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:18.441465 (+794656us) spinlock_profiling.cc:243] Waited 9.13 ms on lock 0x7b48000fc0d8. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f46a631a1 00007f9f46a5a237 00007f9f46a5a0ca 00007f9f46b5192f 00007f9f46b515f7 00007f9f46b65c1c 00007f9f46b65bc8 00007f9f46b65b48 00007f9f46b65b03 00007f9f46b64dac 00007f9f45a8db36
0503 13:58:18.865280 (+423815us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b2c000655f8. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f40425c81 00007f9f404355e2 00007f9f41b66de1 00007f9f41b2b81a 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22
0503 13:58:18.902772 (+ 37492us) log.cc:844] Serialized 26 byte log entry
0503 13:58:19.031414 (+128642us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:19.031454 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:19.031506 (+ 52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:19.631244 (+599738us) raft_consensus.cc:1685] finished
0503 13:58:19.631265 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:19.732302 (+101037us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1223,"raft.run_cpu_time_us":11002,"raft.run_wall_time_us":792992,"spinlock_wait_cycles":28371200,"thread_start_us":91791,"threads_started":1,"wal-append.queue_time_us":165245}
I20260503 13:58:19.783164 10167 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.796216 9898 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.798831 10173 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:19.776227 10191 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.825888 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 239) took 2238 ms. Trace:
I20260503 13:58:19.826483 9643 consensus_queue.cc:260] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.778815 10122 raft_consensus.cc:515] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.792281 10171 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.791179 10163 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.787935 10158 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.795094 10097 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.794222 10100 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.818802 9449 raft_consensus.cc:2393] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:19.830979 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:19.817236 10160 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:19.828127 9451 rpcz_store.cc:276] 0503 13:58:17.613052 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.844397 (+231345us) service_pool.cc:224] Handling call
0503 13:58:17.881212 (+ 36815us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:18.973379 (+1092167us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:19.825725 (+852346us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400303754. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:19.825850 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2386048384}
I20260503 13:58:19.733536 9448 rpcz_store.cc:276] 0503 13:58:17.536082 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.604115 (+ 68033us) service_pool.cc:224] Handling call
0503 13:58:17.604420 (+ 305us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:17.668720 (+ 64300us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:17.668752 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:18.443682 (+774930us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b2c00107268. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f40425c81 00007f9f404355e2 00007f9f41b66de1 00007f9f41b2b81a 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22
0503 13:58:18.444179 (+ 497us) log.cc:844] Serialized 26 byte log entry
0503 13:58:18.885983 (+441804us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:18.886030 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:18.886106 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:19.044799 (+158693us) raft_consensus.cc:1685] finished
0503 13:58:19.044817 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:19.082762 (+ 37945us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53783,"raft.queue_time_us":65104,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":91,"spinlock_wait_cycles":5145344,"thread_start_us":429210,"threads_started":1,"wal-append.queue_time_us":457477}
I20260503 13:58:19.818172 10028 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:19.836484 9526 raft_consensus.cc:1275] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:19.804829 10070 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.824290 10186 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.643303 9450 rpcz_store.cc:276] 0503 13:58:17.614473 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.987549 (+373076us) service_pool.cc:224] Handling call
0503 13:58:17.987847 (+ 298us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:17.994847 (+ 7000us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:17.996378 (+ 1531us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:18.863622 (+867244us) log.cc:844] Serialized 26 byte log entry
0503 13:58:19.018752 (+155130us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:19.020633 (+ 1881us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:19.024952 (+ 4319us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:19.266388 (+241436us) raft_consensus.cc:1685] finished
0503 13:58:19.266411 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:19.266593 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39275,"raft.queue_time_us":4049,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":3233,"thread_start_us":88004,"threads_started":1,"wal-append.queue_time_us":169734}
I20260503 13:58:19.741160 9597 rpcz_store.cc:276] 0503 13:58:17.707508 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.715582 (+ 8074us) service_pool.cc:224] Handling call
0503 13:58:17.724469 (+ 8887us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:17.726125 (+ 1656us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:17.726157 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:17.726601 (+ 444us) log.cc:844] Serialized 26 byte log entry
0503 13:58:19.041675 (+1315074us) spinlock_profiling.cc:243] Waited 3.54 ms on lock 0x7b180023c440. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea70587 00007f9f47869bc9 00007f9f3ea8ccdb 00007f9f3ea8af7f 00007f9f3ea8d242 00007f9f416c45ab 00007f9f416ca0c3 00007f9f416ca471 00007f9f41ae6def
0503 13:58:19.538154 (+496479us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:19.566154 (+ 28000us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:19.633562 (+ 67408us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:19.650517 (+ 16955us) raft_consensus.cc:1685] finished
0503 13:58:19.650550 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:19.650827 (+ 277us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":787,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"spinlock_wait_cycles":8147840,"thread_start_us":242704,"threads_started":1,"wal-append.queue_time_us":1921962}
I20260503 13:58:19.847102 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:19.840507 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:19.848692 9444 raft_consensus.cc:2468] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:19.836731 10039 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:19.829169 10185 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.845326 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:19.815450 10183 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:19.847852 10196 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.760927 9446 rpcz_store.cc:276] 0503 13:58:17.499543 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.525673 (+ 26130us) service_pool.cc:224] Handling call
0503 13:58:17.569505 (+ 43832us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:17.662460 (+ 92955us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:17.666068 (+ 3608us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:19.002968 (+1336900us) spinlock_profiling.cc:243] Waited 1.03 ms on lock 0x7b2c00107268. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f40425c81 00007f9f404355e2 00007f9f41b66de1 00007f9f41b2b81a 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22
0503 13:58:19.025266 (+ 22298us) log.cc:844] Serialized 26 byte log entry
0503 13:58:19.090792 (+ 65526us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:19.094158 (+ 3366us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:19.094234 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:19.659022 (+564788us) raft_consensus.cc:1685] finished
0503 13:58:19.659043 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:19.659279 (+ 236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106592,"raft.queue_time_us":88929,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"spinlock_wait_cycles":2372992,"thread_start_us":58586,"threads_started":1,"wal-append.queue_time_us":625849}
I20260503 13:58:19.837973 10073 raft_consensus.cc:3055] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:19.855415 9451 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:19.838333 10028 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.844014 9898 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:19.865141 10200 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.868381 10037 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:19.851891 9524 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 178) took 2135 ms (client timeout 1529 ms). Trace:
W20260503 13:58:19.876088 10187 log.cc:927] Time spent T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260503 13:58:19.865649 10073 raft_consensus.cc:740] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:19.850018 10186 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:19.851794 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:19.850570 9520 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:19.838922 10160 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.834777 10190 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.834712 10188 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.890276 9519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 261) took 2208 ms. Trace:
I20260503 13:58:19.907091 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:19.861012 10199 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.909045 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:19.909725 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:19.866982 10028 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:19.862043 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:19.864163 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:19.908093 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:19.926657 10184 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:19.867440 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 237) took 2367 ms. Trace:
I20260503 13:58:19.883951 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:19.885663 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:19.884641 10173 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:19.836541 10168 consensus_queue.cc:237] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.877453 10183 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.883250 10196 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:19.876224 10200 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:19.875808 9524 rpcz_store.cc:269] 0503 13:58:17.708662 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.782116 (+ 73454us) service_pool.cc:224] Handling call
0503 13:58:19.839078 (+2056962us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400349c54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:19.842064 (+ 2986us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2511437568}
I20260503 13:58:19.885135 10037 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:20.519593 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:20.522435 9451 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
W20260503 13:58:20.519133 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 198208us
I20260503 13:58:19.935145 10125 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.564646 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:20.564034 9644 raft_consensus.cc:359] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.356523 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:20.457998 9602 consensus_meta.cc:287] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.650s user 0.008s sys 0.000s
I20260503 13:58:20.576221 9451 raft_consensus.cc:2468] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:20.487978 10190 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:19.944416 9528 raft_consensus.cc:2468] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:20.525520 9525 raft_consensus.cc:2468] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:20.525357 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.028204 10122 leader_election.cc:290] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.567642 10171 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.568140 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:20.561501 10173 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.584129 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.584843 9446 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:20.585719 10173 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:20.586194 10173 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:20.569887 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 172) took 1369 ms. Trace:
I20260503 13:58:20.361867 9593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 190) took 1469 ms. Trace:
I20260503 13:58:20.534860 10184 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.974617 10129 leader_election.cc:290] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.163075 10163 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.520753 10185 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.029363 10034 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:20.008422 10026 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.535035 10037 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:20.535445 9566 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.26.193:45599 timed out after 1.726s (SENT)
I20260503 13:58:20.575393 9644 raft_consensus.cc:385] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:20.596168 9644 raft_consensus.cc:740] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:20.586602 10173 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:20.599028 9418 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.813s user 0.023s sys 0.047s
W20260503 13:58:20.599431 9418 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.813s user 0.023s sys 0.047s
I20260503 13:58:20.306917 10159 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:20.307174 9898 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:19.935859 10170 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.246088 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:20.532385 10092 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.708s user 0.003s sys 0.006s
I20260503 13:58:20.344918 10099 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:20.089484 10195 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:19.931102 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:20.571208 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.566929 9950 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.582373 9642 tablet_bootstrap.cc:654] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:20.589542 9593 rpcz_store.cc:276] 0503 13:58:18.892367 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:18.893529 (+ 1162us) service_pool.cc:224] Handling call
0503 13:58:20.350796 (+1457267us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b5800010898. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:20.361375 (+ 10579us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8448768}
I20260503 13:58:20.354826 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:20.583804 9527 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:20.112413 10169 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.181180 10191 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.240064 10070 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.182420 10167 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.534237 9453 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:20.563225 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:19.970175 10201 consensus_queue.cc:1048] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.749s
I20260503 13:58:20.569216 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:20.562798 9523 raft_consensus.cc:2468] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:20.587407 10097 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.587179 9594 rpcz_store.cc:276] 0503 13:58:19.200176 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.838365 (+638189us) service_pool.cc:224] Handling call
0503 13:58:20.569827 (+731462us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:20.592388 10113 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:20.665436 10113 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.584985 9524 raft_consensus.cc:2410] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:20.667119 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.668164 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.621411 9449 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:20.625540 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.593005 10200 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.562736 9448 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:20.593031 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:20.132613 10186 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.596931 9644 consensus_queue.cc:260] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.344697 10039 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:20.578346 9602 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:19.922152 9519 rpcz_store.cc:276] 0503 13:58:17.681495 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.684103 (+ 2608us) service_pool.cc:224] Handling call
0503 13:58:18.985721 (+1301618us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b5800004598. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50a3e 000000000035a3c9 00007f9f46a888e0 00007f9f46b51784 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:18.985873 (+ 152us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:19.081068 (+ 95195us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:19.081100 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:19.081146 (+ 46us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:19.081179 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:19.889991 (+808812us) spinlock_profiling.cc:243] Waited 93.2 ms on lock 0x7b54003feb54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:19.890051 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:19.890197 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":223053952}
I20260503 13:58:20.602873 10185 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:20.603298 9974 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.565258 10199 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:20.591027 9595 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 191) took 1698 ms (client timeout 1829 ms). Trace:
W20260503 13:58:20.605967 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 82210.6us
I20260503 13:58:20.597635 10163 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:20.596058 10184 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:20.610317 10091 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.004s sys 0.018s
I20260503 13:58:20.608498 9898 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.532619 9445 rpcz_store.cc:276] 0503 13:58:17.500466 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.596288 (+ 95822us) service_pool.cc:224] Handling call
0503 13:58:17.613424 (+ 17136us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:17.670619 (+ 57195us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:17.672686 (+ 2067us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:17.716814 (+ 44128us) log.cc:844] Serialized 26 byte log entry
0503 13:58:17.841289 (+124475us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b48000f0558. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f46a631a1 00007f9f46a5a237 00007f9f46a5a0ca 00007f9f46b5192f 00007f9f46b515f7 00007f9f46b65c1c 00007f9f46b65bc8 00007f9f46b65b48 00007f9f46b65b03 00007f9f46b64dac 00007f9f45a8db36
0503 13:58:17.983289 (+142000us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:18.053484 (+ 70195us) log.cc:844] Serialized 14 byte log entry
0503 13:58:18.053698 (+ 214us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:18.053795 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:19.038104 (+984309us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:19.853023 (+814919us) raft_consensus.cc:1685] finished
0503 13:58:19.854435 (+ 1412us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:19.865882 (+ 11447us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":171323,"prepare.queue_time_us":60736,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":64250,"raft.run_cpu_time_us":1126,"raft.run_wall_time_us":167009,"spinlock_wait_cycles":31216768,"thread_start_us":143418,"threads_started":1,"wal-append.queue_time_us":444557}
I20260503 13:58:20.605520 10159 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.610857 10099 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.610374 10170 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:20.611248 10195 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:20.606889 10092 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.782s user 0.004s sys 0.008s
W20260503 13:58:20.609957 10151 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.007s sys 0.018s
I20260503 13:58:20.609702 10129 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:20.645551 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 110130us
I20260503 13:58:20.664932 10204 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:20.752674 9643 ts_tablet_manager.cc:1434] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 1.673s user 0.018s sys 0.030s
I20260503 13:58:20.610253 9450 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:20.672333 9601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 194) took 1774 ms. Trace:
I20260503 13:58:20.852612 9601 rpcz_store.cc:276] 0503 13:58:18.898760 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.053731 (+154971us) service_pool.cc:224] Handling call
0503 13:58:19.270037 (+216306us) spinlock_profiling.cc:243] Waited 3.03 ms on lock 0x7b5800010898. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:20.671619 (+1401582us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b5800010898. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea509ec 00007f9f3ea505ab 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:20.671780 (+ 161us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:20.671942 (+ 162us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:20.671966 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:20.672089 (+ 123us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:20.672122 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:20.672174 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:20.672273 (+ 99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":32443136}
I20260503 13:58:20.860705 10039 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.712037 9447 raft_consensus.cc:2393] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
W20260503 13:58:20.850342 9643 ts_tablet_manager.cc:1467] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868: Tablet startup took 1770ms
W20260503 13:58:20.796416 10166 consensus_meta.cc:287] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.990s user 0.009s sys 0.000s
I20260503 13:58:20.860344 9492 leader_election.cc:304] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:20.702483 9598 consensus_meta.cc:287] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.915s user 0.005s sys 0.004s
W20260503 13:58:20.848997 10151 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.875s user 0.008s sys 0.030s
I20260503 13:58:20.702132 10100 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.858934 9450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 269) took 1022 ms. Trace:
W20260503 13:58:20.869624 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 242318us
I20260503 13:58:20.704021 9446 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:20.869953 9452 raft_consensus.cc:2468] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:20.873636 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 234) took 1882 ms. Trace:
I20260503 13:58:20.710172 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.873980 9418 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:20.669116 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.670670 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.874902 9522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 265) took 1022 ms. Trace:
I20260503 13:58:20.874971 10186 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.875298 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.875532 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 265) took 1049 ms. Trace:
I20260503 13:58:20.669370 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:20.877143 9522 rpcz_store.cc:276] 0503 13:58:19.852766 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.863408 (+ 10642us) service_pool.cc:224] Handling call
0503 13:58:20.670535 (+807127us) spinlock_profiling.cc:243] Waited 3.5 ms on lock 0x7b5800004598. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46aff760 00007f9f46aff044 00007f9f4006c834 00007f9f4006c7c2
0503 13:58:20.874847 (+204312us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8042880}
I20260503 13:58:20.867604 9602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 200) took 1880 ms. Trace:
I20260503 13:58:20.875878 9447 rpcz_store.cc:276] 0503 13:58:19.830733 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.840251 (+ 9518us) service_pool.cc:224] Handling call
0503 13:58:20.875475 (+1035224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:20.875926 9524 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:20.880380 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:20.880849 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.879470 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:20.880973 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:20.881484 9524 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:20.881991 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.882315 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.882594 9519 raft_consensus.cc:2468] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:20.882529 10200 raft_consensus.cc:2749] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:20.873989 9446 rpcz_store.cc:276] 0503 13:58:18.991572 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:20.525106 (+1533534us) service_pool.cc:224] Handling call
0503 13:58:20.873582 (+348476us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:20.883582 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:20.872412 10175 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:20.872768 9450 rpcz_store.cc:276] 0503 13:58:19.833076 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:20.245727 (+412651us) service_pool.cc:224] Handling call
0503 13:58:20.854471 (+608744us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:20.874292 9523 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:20.876935 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:20.859287 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 266) took 1028 ms. Trace:
I20260503 13:58:20.708882 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:20.705005 10110 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:20.855967 10091 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.881s user 0.007s sys 0.026s
I20260503 13:58:20.709558 9973 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.849810 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.873093 10173 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:20.854943 10204 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.883611 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:20.790774 9596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 174) took 1590 ms (client timeout 1594 ms). Trace:
I20260503 13:58:20.674489 10160 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.712577 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:20.871877 9492 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:20.898615 9525 raft_consensus.cc:2368] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3.
I20260503 13:58:20.690914 10183 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:20.712121 10196 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:20.664072 9453 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 238) took 1643 ms (client timeout 2083 ms). Trace:
I20260503 13:58:20.789765 10037 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:20.852007 10182 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:20.662869 9597 raft_consensus.cc:2468] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:20.863184 10185 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.858991 10193 raft_consensus.cc:2764] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260503 13:58:20.898792 9644 ts_tablet_manager.cc:1434] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 1.107s user 0.036s sys 0.041s
W20260503 13:58:20.899101 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 54246.5us
I20260503 13:58:20.881901 9445 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:20.663872 10191 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:20.879185 10184 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.707417 10028 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.877017 9974 raft_consensus.cc:2804] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:20.879240 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:20.902480 9601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 175) took 1371 ms. Trace:
I20260503 13:58:20.906147 9527 raft_consensus.cc:2468] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:20.877653 9598 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
W20260503 13:58:20.800747 9595 rpcz_store.cc:269] 0503 13:58:18.892694 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:18.893497 (+ 803us) service_pool.cc:224] Handling call
0503 13:58:20.354015 (+1460518us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b5800010898. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50a3e 000000000035a3c9 00007f9f46a888e0 00007f9f46b51784 00007f9f46b517f1 00007f9f46aff760 00007f9f46aff044 00007f9f4006c834 00007f9f4006c7c2
0503 13:58:20.589916 (+235901us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7339776}
I20260503 13:58:20.855445 10129 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.894073 9521 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
W20260503 13:58:20.868739 9643 ts_tablet_manager.cc:1469] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868: Trace:
0503 13:58:05.282601 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:05.810412 (+527811us) service_pool.cc:224] Handling call
0503 13:58:05.971160 (+160748us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:05.971250 (+ 90us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.192882 (+221632us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.243713 (+ 50831us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.244101 (+ 388us) inbound_call.cc:177] Queueing success response
0503 13:58:18.100800 (+11856699us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0503 13:58:19.079985 (+979185us) ts_tablet_manager.cc:1435] Starting tablet replica
0503 13:58:19.130068 (+ 50083us) tablet_replica.cc:261] Starting instrumentation
0503 13:58:19.130526 (+ 458us) tablet_replica.cc:280] Starting consensus
0503 13:58:20.667218 (+1536692us) spinlock_profiling.cc:243] Waited 1.36 ms on lock 0x7b18001a0440. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f416d285c 00007f9f3ea70c0b 00007f9f47869bc9 00007f9f3ea8ccdb 00007f9f3ea8af7f 00007f9f3ea8aa95 00007f9f41b23440 00007f9f41b21151 00007f9f45528f9e
Metrics: {"mutex_wait_us":9377,"raft.queue_time_us":22212,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":10976,"spinlock_wait_cycles":3131136,"tablet-open.queue_time_us":11856764,"thread_start_us":31324,"threads_started":1}
I20260503 13:58:20.884407 9418 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:20.906490 9492 leader_election.cc:400] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Vote denied by peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599) with higher term. Message: Invalid argument: T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3.
I20260503 13:58:20.986806 9492 leader_election.cc:403] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260503 13:58:20.893954 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:20.954520 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 44681us
I20260503 13:58:20.898208 9523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 266) took 1042 ms. Trace:
I20260503 13:58:20.898345 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:20.867990 10170 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.868088 10163 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.877382 10166 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.896519 9446 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
W20260503 13:58:20.958200 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 104404us
I20260503 13:58:20.879753 9602 rpcz_store.cc:276] 0503 13:58:18.987442 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.045721 (+ 58279us) service_pool.cc:224] Handling call
0503 13:58:20.863103 (+1817382us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:20.960542 9644 ts_tablet_manager.cc:1467] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6: Tablet startup took 1168ms
I20260503 13:58:20.894311 9448 rpcz_store.cc:276] 0503 13:58:19.831059 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.841800 (+ 10741us) service_pool.cc:224] Handling call
0503 13:58:20.854471 (+1012671us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:20.910512 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 209) took 1100 ms. Trace:
I20260503 13:58:20.991480 9449 raft_consensus.cc:2468] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:20.911593 10110 raft_consensus.cc:2749] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:20.955420 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:21.001257 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.000942 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:21.002225 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:21.002619 9445 raft_consensus.cc:2393] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:21.002840 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:21.003247 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:21.000991 9594 rpcz_store.cc:276] 0503 13:58:19.810510 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:20.712306 (+901796us) service_pool.cc:224] Handling call
0503 13:58:20.910455 (+198149us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.003427 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:20.961150 10200 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:20.959570 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 193) took 1189 ms. Trace:
I20260503 13:58:20.891690 10175 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.008219 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.008486 9525 rpcz_store.cc:276] 0503 13:58:19.766355 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:20.592762 (+826407us) service_pool.cc:224] Handling call
0503 13:58:20.956169 (+363407us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.009358 9520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 264) took 1159 ms. Trace:
I20260503 13:58:21.009495 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.010046 9599 raft_consensus.cc:2393] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:21.010109 9525 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:20.904255 10213 consensus_queue.cc:1059] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
W20260503 13:58:20.991133 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 84765.8us
I20260503 13:58:21.011340 9492 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:20.991668 9567 leader_election.cc:343] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:20.962949 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 176) took 1319 ms. Trace:
W20260503 13:58:20.994139 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 109900us
I20260503 13:58:20.980706 10165 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:20.980517 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:20.905520 9596 rpcz_store.cc:269] 0503 13:58:19.207594 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.559167 (+351573us) service_pool.cc:224] Handling call
0503 13:58:20.790587 (+1231420us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54000ab754. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:20.790708 (+ 121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":242404608}
I20260503 13:58:21.015728 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.016346 9596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 211) took 1180 ms. Trace:
I20260503 13:58:21.016686 9596 rpcz_store.cc:276] 0503 13:58:19.836240 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.015401 (+1179161us) service_pool.cc:224] Handling call
0503 13:58:21.016293 (+ 892us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.017611 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.018280 9596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 216) took 1042 ms. Trace:
I20260503 13:58:21.018590 9596 rpcz_store.cc:276] 0503 13:58:19.975454 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.017353 (+1041899us) service_pool.cc:224] Handling call
0503 13:58:21.018237 (+ 884us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:20.995693 10113 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:20.990706 10199 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.991391 9974 raft_consensus.cc:697] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:20.997946 9528 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:21.022869 10165 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.009732 9520 rpcz_store.cc:276] 0503 13:58:19.850304 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.008575 (+1158271us) service_pool.cc:224] Handling call
0503 13:58:21.008864 (+ 289us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:21.009034 (+ 170us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:21.009072 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:21.009116 (+ 44us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:21.009155 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.009210 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:21.009307 (+ 97us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.002024 9602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 178) took 1186 ms. Trace:
I20260503 13:58:20.984685 10039 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.981192 9601 rpcz_store.cc:276] 0503 13:58:19.531286 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:20.860300 (+1329014us) service_pool.cc:224] Handling call
0503 13:58:20.902323 (+ 42023us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:20.991322 9598 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 171) took 1791 ms (client timeout 1821 ms). Trace:
I20260503 13:58:20.999578 9593 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:21.013481 9492 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.007201 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.038461 9492 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:21.037709 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.038870 10165 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.038475 9593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 206) took 1597 ms. Trace:
I20260503 13:58:21.039341 9492 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.020201 10110 raft_consensus.cc:2749] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:21.039410 9593 rpcz_store.cc:276] 0503 13:58:19.440944 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:20.851545 (+1410601us) service_pool.cc:224] Handling call
0503 13:58:21.038414 (+186869us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.013064 10200 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.033648 10195 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:21.040513 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.041153 10170 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.035722 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:21.042291 9448 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
W20260503 13:58:21.034117 9598 rpcz_store.cc:269] 0503 13:58:19.199893 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.637498 (+437605us) service_pool.cc:224] Handling call
0503 13:58:20.987482 (+1349984us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.043099 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:21.013521 9567 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:21.014287 9418 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.038962 10028 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.044075 9595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 207) took 1533 ms. Trace:
I20260503 13:58:21.044418 9595 rpcz_store.cc:276] 0503 13:58:19.510766 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:20.997974 (+1487208us) service_pool.cc:224] Handling call
0503 13:58:21.044008 (+ 46034us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.044520 9418 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:21.045359 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.045589 9418 leader_election.cc:304] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:21.046422 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.046660 10196 raft_consensus.cc:2804] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.047012 10196 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.047389 10196 raft_consensus.cc:3060] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.047441 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.048635 9418 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:20.960800 10182 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.013235 9450 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:21.007772 10158 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:20.970649 10191 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:20.999782 9644 ts_tablet_manager.cc:1469] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6: Trace:
0503 13:58:05.278845 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.112265 (+833420us) service_pool.cc:224] Handling call
0503 13:58:06.159393 (+ 47128us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.159481 (+ 88us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.203139 (+ 43658us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.285723 (+ 82584us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.286138 (+ 415us) inbound_call.cc:177] Queueing success response
0503 13:58:17.829844 (+11543706us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0503 13:58:19.791933 (+1962089us) ts_tablet_manager.cc:1435] Starting tablet replica
0503 13:58:19.876138 (+ 84205us) tablet_replica.cc:261] Starting instrumentation
0503 13:58:20.519541 (+643403us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":22069,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":115,"tablet-open.queue_time_us":11543764}
I20260503 13:58:21.020465 9643 tablet_bootstrap.cc:492] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:21.021067 9566 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:21.003371 9446 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:21.022387 10185 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.025976 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:21.009519 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.062438 10204 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:21.019626 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:20.997547 10184 raft_consensus.cc:2804] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.024722 9491 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:21.006625 9444 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:21.025770 9974 consensus_queue.cc:237] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:20.994232 10160 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
W20260503 13:58:21.025453 9600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 196) took 2105 ms (client timeout 1626 ms). Trace:
I20260503 13:58:20.990309 9523 rpcz_store.cc:276] 0503 13:58:19.854458 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:20.709919 (+855461us) service_pool.cc:224] Handling call
0503 13:58:20.896360 (+186441us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.033088 10159 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.034577 10129 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.034780 9447 raft_consensus.cc:2468] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:21.035964 9522 raft_consensus.cc:2410] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260503 13:58:21.071113 9452 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 256) took 2890 ms (client timeout 1726 ms). Trace:
I20260503 13:58:21.036262 10099 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.035619 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.053210 10199 consensus_queue.cc:1048] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260503 13:58:21.071861 10191 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.072151 9601 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:21.009747 10166 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:20.959837 9453 rpcz_store.cc:269] 0503 13:58:19.021103 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.837017 (+815914us) service_pool.cc:224] Handling call
0503 13:58:20.650405 (+813388us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.053781 9418 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:21.052911 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 264) took 1252 ms. Trace:
I20260503 13:58:21.055099 9644 tablet_bootstrap.cc:492] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:21.022434 10163 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.039628 10039 raft_consensus.cc:2804] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260503 13:58:21.061000 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:21.043560 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.065492 9520 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:21.065935 9349 rpcz_store.cc:275] Call kudu.master.MasterService.IsCreateTableDone from 127.0.0.1:51412 (request call id 38) took 1539 ms. Trace:
I20260503 13:58:21.013827 9597 rpcz_store.cc:276] 0503 13:58:19.640998 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:20.568965 (+927967us) service_pool.cc:224] Handling call
0503 13:58:20.960452 (+391487us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:21.064543 9491 leader_election.cc:343] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:21.039500 10165 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260503 13:58:21.067054 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 46034.6us
W20260503 13:58:21.067148 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 53699.5us
I20260503 13:58:21.065815 10171 raft_consensus.cc:2804] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:21.067802 10195 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:21.041457 10200 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.043628 9448 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:21.089087 10195 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:21.071811 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:21.041110 9593 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:21.053067 10079 raft_consensus.cc:2804] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:21.062964 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:21.053387 10170 raft_consensus.cc:2804] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.054247 9991 raft_consensus.cc:2804] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.065636 10073 consensus_queue.cc:260] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.027113 9602 rpcz_store.cc:276] 0503 13:58:19.815873 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.001018 (+1185145us) service_pool.cc:224] Handling call
0503 13:58:21.001967 (+ 949us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:21.068477 9600 rpcz_store.cc:269] 0503 13:58:18.924597 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.267191 (+342594us) service_pool.cc:224] Handling call
0503 13:58:20.891745 (+1624554us) spinlock_profiling.cc:243] Waited 63.9 ms on lock 0x7b5800010898. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea50adc 00007f9f3ea50a46 000000000035a3c9 00007f9f46a888e0 00007f9f46b51784 00007f9f46b517f1 00007f9f46aff760 00007f9f46aff044 00007f9f4006c834 00007f9f4006c7c2
0503 13:58:21.023491 (+131746us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":147125248}
I20260503 13:58:21.074069 9594 raft_consensus.cc:2368] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3.
I20260503 13:58:21.069123 10182 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:21.055593 10168 raft_consensus.cc:2749] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:21.075922 9418 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:21.076403 10039 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.070284 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.085235 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:21.086376 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.066546 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.066134 10184 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.088080 9566 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:21.088133 9567 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
W20260503 13:58:21.227715 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 139650us
I20260503 13:58:21.227923 9601 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:21.207281 9642 tablet_bootstrap.cc:492] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:21.089475 10195 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
W20260503 13:58:21.228591 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 152690us
W20260503 13:58:21.231072 9418 leader_election.cc:343] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:21.234701 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 221) took 3550 ms. Trace:
I20260503 13:58:21.231592 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.237178 9451 rpcz_store.cc:276] 0503 13:58:17.684023 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.228371 (+3544348us) service_pool.cc:224] Handling call
0503 13:58:21.234254 (+ 5883us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:21.234406 (+ 152us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:21.234429 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:21.234469 (+ 40us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:21.234497 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.234550 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:21.234648 (+ 98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.238236 9351 catalog_manager.cc:5671] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:21.239027 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 222) took 3530 ms. Trace:
I20260503 13:58:21.239387 9451 rpcz_store.cc:276] 0503 13:58:17.708634 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.238364 (+3529730us) service_pool.cc:224] Handling call
0503 13:58:21.238629 (+ 265us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:21.238765 (+ 136us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:21.238786 (+ 21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:21.238821 (+ 35us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:21.238846 (+ 25us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.238891 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:21.238981 (+ 90us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.240269 9520 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:21.240103 9351 catalog_manager.cc:5671] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } }
I20260503 13:58:21.241588 9351 catalog_manager.cc:5671] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:21.074729 10227 raft_consensus.cc:2804] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.091138 9491 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:21.243549 9351 catalog_manager.cc:5671] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } }
I20260503 13:58:21.237396 9418 leader_election.cc:400] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Vote denied by peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805) with higher term. Message: Invalid argument: T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3.
W20260503 13:58:21.244494 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 153389us
I20260503 13:58:21.244863 9418 leader_election.cc:403] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260503 13:58:21.247149 9491 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:21.076988 10191 raft_consensus.cc:2804] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:21.229272 9642 ts_tablet_manager.cc:1403] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 2.237s user 0.029s sys 0.063s
I20260503 13:58:21.230472 9567 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:21.085083 9349 rpcz_store.cc:276] 0503 13:58:19.528767 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.528983 (+ 216us) service_pool.cc:224] Handling call
0503 13:58:19.529344 (+ 361us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0503 13:58:19.529522 (+ 178us) catalog_manager.cc:2291] Verify if the table creation is in progress for test-workload [id=54209d96daad4531bdc33057c81f7430]
0503 13:58:21.056181 (+1526659us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1525919}
W20260503 13:58:21.227958 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 139958us
I20260503 13:58:21.229260 10166 raft_consensus.cc:2804] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:21.137092 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.229286 10202 raft_consensus.cc:2804] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.253876 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.229689 9991 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.244225 10227 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:21.089102 10171 raft_consensus.cc:697] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:21.255355 9644 tablet_bootstrap.cc:654] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:21.255707 9643 tablet_bootstrap.cc:654] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:21.190521 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.244880 9351 catalog_manager.cc:5671] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 1 to 2, leader changed from 1cb63886040047559d3893923cc9aaf6 (127.8.26.193) to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 2 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:21.229861 10184 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.161936 10188 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.227563 10079 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.074925 9446 rpcz_store.cc:276] 0503 13:58:19.807778 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.002592 (+1194814us) service_pool.cc:224] Handling call
0503 13:58:21.052856 (+ 50264us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.123667 10200 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:21.291193 10200 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.285408 9991 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.292964 10200 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.089498 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.079146 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:21.162384 9448 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:21.230281 10039 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.234191 10193 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:21.142745 10175 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.228549 10165 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:21.227880 10170 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.252017 10185 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.260215 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.261641 10160 raft_consensus.cc:2749] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599) with higher term. Message: Invalid argument: T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3.
I20260503 13:58:21.253505 10191 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.261899 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.308579 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.163511 9444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 267) took 1332 ms. Trace:
I20260503 13:58:21.132377 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:21.263162 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 248) took 3592 ms. Trace:
I20260503 13:58:21.264792 9524 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:21.265952 9566 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:21.313571 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 47685.9us
I20260503 13:58:21.314224 9566 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:21.314975 9566 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:21.251920 10199 raft_consensus.cc:2749] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805) with higher term. Message: Invalid argument: T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3.
I20260503 13:58:21.241603 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.317529 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.317824 9642 raft_consensus.cc:359] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.318434 9642 raft_consensus.cc:385] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:21.318559 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.319140 9520 raft_consensus.cc:2468] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:21.319950 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.256352 10166 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:21.267728 10195 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:21.266680 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 215893us
I20260503 13:58:21.321964 10195 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:21.322136 9492 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.322520 10195 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:21.322824 10195 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
W20260503 13:58:21.323079 9492 leader_election.cc:343] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:21.323163 10195 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:21.323695 9492 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
W20260503 13:58:21.269901 9599 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 203) took 2061 ms (client timeout 2083 ms). Trace:
I20260503 13:58:21.255025 10129 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:21.271559 10179 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:21.273483 9898 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.273999 9567 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:21.324455 10129 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.307466 10185 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.325394 10017 raft_consensus.cc:2804] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:21.325845 10017 raft_consensus.cc:697] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:21.271463 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.276001 10091 raft_consensus.cc:3055] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:21.326480 10185 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.277637 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 249) took 3491 ms. Trace:
I20260503 13:58:21.326808 10091 raft_consensus.cc:740] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:21.278584 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.278162 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 273) took 1257 ms. Trace:
I20260503 13:58:21.313262 9593 raft_consensus.cc:2468] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:21.327996 10129 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.284646 10171 consensus_queue.cc:237] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.328588 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.329723 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.330828 9491 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:21.234485 9595 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:21.326972 9447 rpcz_store.cc:276] 0503 13:58:17.814770 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.276967 (+3462197us) service_pool.cc:224] Handling call
0503 13:58:21.277248 (+ 281us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:21.277383 (+ 135us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:21.277409 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:21.277448 (+ 39us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:21.277478 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.277517 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:21.277598 (+ 81us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.327409 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.324505 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.320916 10224 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:21.332219 9566 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:21.286531 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.347635 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 224) took 3454 ms. Trace:
I20260503 13:58:21.346984 10179 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:21.348194 10179 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:21.346711 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.348580 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.347010 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.349257 9596 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:21.349519 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.349880 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.350100 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.350358 10224 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:21.320658 10099 raft_consensus.cc:2804] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:21.351161 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.351447 10224 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:21.351497 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.352404 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:21.352650 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:21.353323 9600 raft_consensus.cc:2368] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4.
I20260503 13:58:21.353395 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.354514 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.348038 9448 rpcz_store.cc:276] 0503 13:58:17.892887 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.309346 (+3416459us) service_pool.cc:224] Handling call
0503 13:58:21.347147 (+ 37801us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:21.347317 (+ 170us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:21.347344 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:21.347387 (+ 43us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:21.347423 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.347484 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:21.347583 (+ 99us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.355546 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.356051 9596 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:21.288444 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.284083 10227 raft_consensus.cc:3060] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.252532 10168 raft_consensus.cc:2804] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:21.257937 10202 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:21.258251 10182 raft_consensus.cc:2764] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260503 13:58:21.075675 9452 rpcz_store.cc:269] 0503 13:58:18.277907 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.196041 (+918134us) service_pool.cc:224] Handling call
0503 13:58:20.354540 (+1158499us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b5800000998. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46aff760 00007f9f46aff044 00007f9f4006c834 00007f9f4006c7c2
0503 13:58:21.070885 (+716345us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b54001a0c54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:21.070989 (+ 104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":583854720}
I20260503 13:58:21.294044 10079 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:21.250814 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 179) took 1432 ms. Trace:
W20260503 13:58:21.311604 9418 leader_election.cc:343] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:21.309484 9444 rpcz_store.cc:276] 0503 13:58:19.831355 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:20.708597 (+877242us) service_pool.cc:224] Handling call
0503 13:58:21.163320 (+454723us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b540039d5d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:21.163443 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":239218048}
I20260503 13:58:21.369519 9444 raft_consensus.cc:1275] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:21.369943 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.370087 9444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 226) took 3366 ms. Trace:
I20260503 13:58:21.370534 9602 raft_consensus.cc:2468] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:21.370486 9444 rpcz_store.cc:276] 0503 13:58:18.003229 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.368844 (+3365615us) service_pool.cc:224] Handling call
0503 13:58:21.369182 (+ 338us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:21.369852 (+ 670us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.370031 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.371017 9594 rpcz_store.cc:276] 0503 13:58:19.817946 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.004021 (+1186075us) service_pool.cc:224] Handling call
0503 13:58:21.243971 (+239950us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.303887 10193 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:21.371945 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.305240 10170 raft_consensus.cc:3060] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.364445 9523 raft_consensus.cc:2468] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:21.372741 10224 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.373472 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.373996 9523 raft_consensus.cc:2468] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:21.374558 10224 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:21.374796 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.375322 9523 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:21.376078 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.318701 9642 raft_consensus.cc:740] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:21.376296 10039 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.327893 9525 raft_consensus.cc:2468] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:21.377148 9642 consensus_queue.cc:260] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.379292 9642 ts_tablet_manager.cc:1434] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.127s user 0.005s sys 0.001s
I20260503 13:58:21.320729 10166 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:21.305426 10165 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:21.275671 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.283376 9416 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:21.347116 10188 raft_consensus.cc:2804] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.346060 9898 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:21.354059 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.282188 10073 raft_consensus.cc:3060] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.286969 9351 catalog_manager.cc:5671] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:21.435403 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.435762 9351 catalog_manager.cc:5671] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } }
I20260503 13:58:21.436465 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.436967 9351 catalog_manager.cc:5671] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 1 to 2, leader changed from 428b342dc4f041df956015e7d7cd173a (127.8.26.194) to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 2 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } }
I20260503 13:58:21.437570 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.438625 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.438306 9351 catalog_manager.cc:5671] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 3 to 4, leader changed from 4bea375a55034750a1f8804b0db97868 (127.8.26.195) to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 4 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:21.439669 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.439960 9351 catalog_manager.cc:5671] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:21.310156 10191 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:21.361066 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 223) took 3577 ms. Trace:
I20260503 13:58:21.331188 9492 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.441378 9351 catalog_manager.cc:5671] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 1 to 2, leader changed from 1cb63886040047559d3893923cc9aaf6 (127.8.26.193) to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 2 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } }
W20260503 13:58:21.431733 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 148408us
W20260503 13:58:21.364846 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 53259.5us
I20260503 13:58:21.365437 9491 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.366391 10184 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.346010 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.372030 10193 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
W20260503 13:58:21.324939 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 51015.2us
I20260503 13:58:21.326725 9527 raft_consensus.cc:2468] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:21.350571 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.348614 10179 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:21.451733 9351 catalog_manager.cc:5671] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 2 to 3, leader changed from 1cb63886040047559d3893923cc9aaf6 (127.8.26.193) to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 3 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } }
I20260503 13:58:21.361948 10202 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.364229 10208 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:21.442667 9451 rpcz_store.cc:276] 0503 13:58:17.783689 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.240151 (+3456462us) service_pool.cc:224] Handling call
0503 13:58:21.240419 (+ 268us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:21.242840 (+ 2421us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:21.257141 (+ 14301us) log.cc:844] Serialized 14 byte log entry
0503 13:58:21.360792 (+103651us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:21.360835 (+ 43us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:21.360872 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.360928 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:21.361022 (+ 94us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":14171,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":65,"spinlock_wait_cycles":5049600,"thread_start_us":103369,"threads_started":1}
I20260503 13:58:21.359897 10168 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:21.452910 9416 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.453897 9642 tablet_bootstrap.cc:492] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
W20260503 13:58:21.454712 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 123551us
I20260503 13:58:21.479211 9521 raft_consensus.cc:2410] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:21.492897 9492 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.362038 10160 raft_consensus.cc:2804] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.453402 9566 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:21.493801 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 50329.1us
I20260503 13:58:21.346704 9519 raft_consensus.cc:2468] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
W20260503 13:58:21.324318 9599 rpcz_store.cc:269] 0503 13:58:19.208343 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.639566 (+431223us) service_pool.cc:224] Handling call
0503 13:58:21.269722 (+1630156us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400038254. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:21.269846 (+ 124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3149427072}
I20260503 13:58:21.430517 10188 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
W20260503 13:58:21.481514 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 116100us
I20260503 13:58:21.482151 9595 raft_consensus.cc:2468] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:21.361588 9991 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.488252 9974 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:21.429621 9898 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.327893 9528 rpcz_store.cc:276] 0503 13:58:20.021140 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:20.598734 (+577594us) service_pool.cc:224] Handling call
0503 13:58:21.278013 (+679279us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54002067d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:21.278107 (+ 94us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":332448640}
I20260503 13:58:21.488703 9567 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:21.490338 9351 catalog_manager.cc:5671] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 1 to 2, leader changed from 1cb63886040047559d3893923cc9aaf6 (127.8.26.193) to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 2 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } }
I20260503 13:58:21.451530 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:21.493893 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 50247.6us
I20260503 13:58:21.508929 9351 catalog_manager.cc:5671] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 2 to 3, leader changed from 1cb63886040047559d3893923cc9aaf6 (127.8.26.193) to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 3 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } }
I20260503 13:58:21.455946 9418 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:21.495850 9492 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.327638 10091 consensus_queue.cc:260] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.486999 9523 raft_consensus.cc:2410] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260503 13:58:21.497685 9416 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:21.326588 10017 consensus_queue.cc:237] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.443517 9522 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:21.488024 10159 raft_consensus.cc:2804] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:21.442593 10185 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260503 13:58:21.499050 9491 leader_election.cc:343] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:21.503772 9644 tablet_bootstrap.cc:492] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:21.508111 9445 raft_consensus.cc:2393] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:21.490571 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.506968 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:21.511312 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 55429.6us
I20260503 13:58:21.511482 9492 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:21.494824 10199 raft_consensus.cc:2749] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:21.495080 10168 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:21.494961 10160 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:21.499963 10163 raft_consensus.cc:2804] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.497174 10089 raft_consensus.cc:2804] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:21.511749 9416 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.505749 10208 raft_consensus.cc:697] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:21.517213 9351 catalog_manager.cc:5671] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:21.500913 10188 raft_consensus.cc:3060] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.507545 9898 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:21.520606 9644 ts_tablet_manager.cc:1403] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.466s user 0.008s sys 0.008s
I20260503 13:58:21.522318 9491 leader_election.cc:400] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Vote denied by peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997) with higher term. Message: Invalid argument: T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4.
W20260503 13:58:21.522442 9492 leader_election.cc:343] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:21.522650 9418 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:21.519804 9567 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:21.312389 9453 rpcz_store.cc:276] 0503 13:58:17.673894 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.091299 (+3417405us) service_pool.cc:224] Handling call
0503 13:58:21.091649 (+ 350us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:21.132414 (+ 40765us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:21.132445 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:21.132921 (+ 476us) log.cc:844] Serialized 26 byte log entry
0503 13:58:21.236794 (+103873us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:21.248705 (+ 11911us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.252885 (+ 4180us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:21.262951 (+ 10066us) raft_consensus.cc:1685] finished
0503 13:58:21.262978 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:21.263100 (+ 122us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1062,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"thread_start_us":95784,"threads_started":1,"wal-append.queue_time_us":125497}
I20260503 13:58:21.524312 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.520300 10185 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:21.518587 10159 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.506161 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.528699 9416 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:21.525815 9452 raft_consensus.cc:1240] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:21.529757 10089 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:21.529919 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 251) took 3646 ms. Trace:
I20260503 13:58:21.527307 10163 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.530743 10196 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.566776 9492 leader_election.cc:304] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.567406 9491 leader_election.cc:403] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260503 13:58:21.519657 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.531698 10199 raft_consensus.cc:2749] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260503 13:58:21.567785 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 45170.8us
W20260503 13:58:21.577044 10230 log.cc:927] Time spent T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260503 13:58:21.572701 9643 tablet_bootstrap.cc:492] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:21.626907 9643 ts_tablet_manager.cc:1403] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.622s user 0.017s sys 0.004s
W20260503 13:58:21.589469 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 69739.7us
I20260503 13:58:21.569526 9991 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.525719 10171 raft_consensus.cc:2764] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260503 13:58:21.520255 10182 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:21.517506 10201 raft_consensus.cc:2804] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:21.517618 10175 raft_consensus.cc:2804] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.517182 10222 raft_consensus.cc:2804] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:21.527390 10160 raft_consensus.cc:3060] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.536708 9351 catalog_manager.cc:5671] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:21.543467 9595 raft_consensus.cc:1240] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:21.576948 10039 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.624598 9418 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:21.524716 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:21.517230 10028 raft_consensus.cc:2804] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.590536 10185 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:21.573165 10184 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.532099 10099 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:21.594933 10159 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:21.658830 10208 consensus_queue.cc:237] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:21.659014 10231 log.cc:927] Time spent T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260503 13:58:21.605842 10089 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:21.578715 10079 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.568775 10200 raft_consensus.cc:2804] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.594377 10204 raft_consensus.cc:2804] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260503 13:58:21.666085 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 144786us
I20260503 13:58:21.660701 10182 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.629462 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:21.629062 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.611348 10195 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.669534 9418 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:21.637274 10175 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.666260 9525 raft_consensus.cc:2410] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260503 13:58:21.665259 10171 raft_consensus.cc:2749] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:21.666851 9991 raft_consensus.cc:2804] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.663522 10201 raft_consensus.cc:697] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:21.675079 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:21.678261 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 155841us
W20260503 13:58:21.678646 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 150015us
I20260503 13:58:21.673053 9450 raft_consensus.cc:2393] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:21.670253 10028 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.623340 10163 raft_consensus.cc:3060] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.672873 10199 raft_consensus.cc:2804] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:21.672700 10099 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:21.671612 10222 raft_consensus.cc:697] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:21.723706 10199 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:21.674965 10039 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:21.726178 10227 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.726752 10222 consensus_queue.cc:237] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.728912 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 253) took 3739 ms. Trace:
W20260503 13:58:21.723402 9565 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:21.729861 9601 raft_consensus.cc:2468] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:21.684264 9418 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:21.731115 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 46909.5us
I20260503 13:58:21.731771 9418 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:21.667128 10166 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.675079 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.685230 9491 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:21.680099 9595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 201) took 2470 ms. Trace:
I20260503 13:58:21.691426 9416 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:21.737444 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 46083.2us
W20260503 13:58:21.738616 9416 leader_election.cc:343] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:21.681316 10200 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.681803 10204 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:21.687191 9991 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:21.690322 10238 raft_consensus.cc:2804] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:21.697548 10079 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.697890 9644 raft_consensus.cc:359] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.700465 9643 raft_consensus.cc:359] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.748715 9643 raft_consensus.cc:385] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:21.748994 9643 raft_consensus.cc:740] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:21.749630 9643 consensus_queue.cc:260] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:21.701531 10237 log.cc:927] Time spent T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260503 13:58:21.708482 10028 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.709985 10201 consensus_queue.cc:237] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:21.711717 9974 raft_consensus.cc:2749] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:21.712117 9642 tablet_bootstrap.cc:654] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:21.712342 9453 raft_consensus.cc:2393] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:21.715875 10195 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:21.605481 9446 rpcz_store.cc:276] 0503 13:58:17.895368 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.289471 (+3394103us) service_pool.cc:224] Handling call
0503 13:58:21.289802 (+ 331us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:21.290113 (+ 311us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:21.290141 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:21.290602 (+ 461us) log.cc:844] Serialized 26 byte log entry
0503 13:58:21.311954 (+ 21352us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:21.428427 (+116473us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.433015 (+ 4588us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:21.527104 (+ 94089us) raft_consensus.cc:1685] finished
0503 13:58:21.527125 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:21.529862 (+ 2737us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3351,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":16644,"threads_started":1,"wal-append.queue_time_us":140315}
I20260503 13:58:21.746546 10200 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.717111 9898 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.717111 10196 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.717830 10182 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:21.684820 10175 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
W20260503 13:58:21.733371 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 48627.9us
I20260503 13:58:21.691224 9492 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:21.777535 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 86368us
I20260503 13:58:21.739665 9416 leader_election.cc:304] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.713831 10159 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.726665 10199 raft_consensus.cc:3060] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:21.748474 9644 raft_consensus.cc:385] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:21.779223 9644 raft_consensus.cc:740] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:21.779843 9644 consensus_queue.cc:260] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.751955 10240 raft_consensus.cc:493] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:21.752148 9991 raft_consensus.cc:3060] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.752645 10079 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.719084 9594 raft_consensus.cc:2468] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:21.729241 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:21.759079 10188 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.759236 10165 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.761858 10170 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.746048 10241 raft_consensus.cc:2804] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.747646 10239 raft_consensus.cc:2804] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:21.765174 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:21.736402 10073 raft_consensus.cc:2804] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.747853 10204 raft_consensus.cc:3060] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.737753 10227 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:21.734354 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 252) took 1092 ms. Trace:
I20260503 13:58:21.745664 10129 raft_consensus.cc:2804] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:21.813289 9566 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:21.748176 10238 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:21.797293 10249 consensus_queue.cc:1048] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260503 13:58:21.798465 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 59874.5us
I20260503 13:58:21.794611 10171 raft_consensus.cc:2804] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:21.812758 9526 raft_consensus.cc:2393] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:21.788532 10184 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:21.817308 9416 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:21.766222 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 267) took 1901 ms. Trace:
I20260503 13:58:21.736661 9595 rpcz_store.cc:276] 0503 13:58:19.207891 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.502608 (+2294717us) service_pool.cc:224] Handling call
0503 13:58:21.518545 (+ 15937us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:21.670806 (+152261us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.678090 (+ 7284us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:21.821195 9416 leader_election.cc:343] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:21.802927 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.801328 10241 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:21.822928 9416 leader_election.cc:304] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.806296 9527 raft_consensus.cc:3060] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.825663 10170 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.822640 9525 rpcz_store.cc:276] 0503 13:58:19.864669 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.725572 (+1860903us) service_pool.cc:224] Handling call
0503 13:58:21.765715 (+ 40143us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:21.765889 (+ 174us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:21.765916 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:21.765959 (+ 43us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:21.765994 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.766052 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:21.766166 (+ 114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:21.828263 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 199) took 2854 ms. Trace:
I20260503 13:58:21.801131 10239 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:21.775910 10193 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.793977 10079 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.822265 10171 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:21.803478 10073 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.813695 10129 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:21.824694 9600 raft_consensus.cc:1240] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:21.842854 9643 ts_tablet_manager.cc:1434] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.216s user 0.012s sys 0.010s
I20260503 13:58:21.846186 10163 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.794507 10240 raft_consensus.cc:515] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.828436 10185 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.853617 10240 leader_election.cc:290] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.853788 10185 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.819309 10238 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:21.834089 9524 raft_consensus.cc:2393] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:21.839025 9520 raft_consensus.cc:2393] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:21.824326 10017 raft_consensus.cc:3060] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:21.834410 9898 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.829497 10250 raft_consensus.cc:2749] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:21.837482 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:21.837606 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 252) took 3926 ms. Trace:
I20260503 13:58:21.830976 10241 raft_consensus.cc:3060] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.883762 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 274) took 1350 ms. Trace:
I20260503 13:58:21.884117 9526 rpcz_store.cc:276] 0503 13:58:20.533392 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.136835 (+603443us) service_pool.cc:224] Handling call
0503 13:58:21.883607 (+746772us) spinlock_profiling.cc:243] Waited 556 ms on lock 0x7b540004db54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:21.883709 (+ 102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1278750976}
I20260503 13:58:21.886809 10193 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:21.886883 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:21.887614 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:21.887904 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:21.850610 9974 consensus_queue.cc:1059] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
W20260503 13:58:21.890215 9521 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 187) took 2127 ms (client timeout 2148 ms). Trace:
I20260503 13:58:21.890154 9991 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:21.890655 9521 rpcz_store.cc:269] 0503 13:58:19.754682 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.760959 (+ 6277us) service_pool.cc:224] Handling call
0503 13:58:19.908886 (+147927us) spinlock_profiling.cc:243] Waited 4.32 ms on lock 0x7b5800004598. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46aff760 00007f9f46aff044 00007f9f4006c834 00007f9f4006c7c2
0503 13:58:21.872642 (+1963756us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b54003fe154. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:21.882008 (+ 9366us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":861573632}
I20260503 13:58:21.891484 9451 raft_consensus.cc:2393] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:21.891767 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:21.892297 9521 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.893078 9418 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.893539 10188 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.894050 10254 raft_consensus.cc:2749] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:21.894323 9991 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.894846 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:21.857244 9644 ts_tablet_manager.cc:1434] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.290s user 0.008s sys 0.014s
I20260503 13:58:21.851764 9452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 255) took 3752 ms. Trace:
I20260503 13:58:21.839147 10239 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:21.847746 10196 raft_consensus.cc:2804] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:21.849936 10170 raft_consensus.cc:2749] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:21.813525 9445 rpcz_store.cc:276] 0503 13:58:20.641819 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.003189 (+361370us) service_pool.cc:224] Handling call
0503 13:58:21.730338 (+727149us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b540017a554. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:21.730470 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1112344320}
I20260503 13:58:21.850666 10073 raft_consensus.cc:3060] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.881553 10165 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.829286 10184 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.882402 9450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 250) took 1281 ms. Trace:
I20260503 13:58:21.874085 9643 tablet_bootstrap.cc:492] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:21.849278 10171 raft_consensus.cc:3060] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:21.875430 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:21.870886 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.880581 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:21.881552 10163 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.857868 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.882701 9453 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:21.888497 10202 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.913012 9600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 210) took 2102 ms. Trace:
I20260503 13:58:21.889521 10166 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:21.916074 9492 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:21.855460 10129 raft_consensus.cc:3060] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:21.911955 10188 raft_consensus.cc:3055] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:21.888429 10160 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.917886 9644 tablet_bootstrap.cc:492] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:21.871546 10168 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.920791 9566 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:21.876781 10028 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:21.913455 9642 tablet_bootstrap.cc:492] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:21.908958 10196 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:21.935624 9642 ts_tablet_manager.cc:1403] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.493s user 0.010s sys 0.009s
I20260503 13:58:21.933326 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:21.938239 10028 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.939517 9491 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:21.729255 9448 rpcz_store.cc:276] 0503 13:58:18.003567 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.355603 (+3352036us) service_pool.cc:224] Handling call
0503 13:58:21.355922 (+ 319us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:21.356245 (+ 323us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:21.356275 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:21.356739 (+ 464us) log.cc:844] Serialized 26 byte log entry
0503 13:58:21.518310 (+161571us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:21.574394 (+ 56084us) log.cc:844] Serialized 14 byte log entry
0503 13:58:21.672236 (+ 97842us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.674427 (+ 2191us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:21.728717 (+ 54290us) raft_consensus.cc:1685] finished
0503 13:58:21.728735 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:21.728859 (+ 124us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":53160,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":85472,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":64933,"thread_start_us":251028,"threads_started":2,"wal-append.queue_time_us":308272}
I20260503 13:58:21.834669 9598 rpcz_store.cc:276] 0503 13:58:18.973468 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.496308 (+2522840us) service_pool.cc:224] Handling call
0503 13:58:21.827591 (+331283us) spinlock_profiling.cc:243] Waited 5.02 ms on lock 0x7b5800010898. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:21.827777 (+ 186us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:21.827941 (+ 164us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:21.827969 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:21.828009 (+ 40us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:21.828045 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.828101 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:21.828197 (+ 96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11539200}
I20260503 13:58:21.922652 10159 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.927712 10224 raft_consensus.cc:493] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:21.977119 9566 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:21.980571 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:21.927500 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:21.931849 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:21.930472 10188 raft_consensus.cc:740] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:21.938810 10196 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:22.014734 10091 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:22.016950 9528 raft_consensus.cc:2393] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:22.018934 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:21.984421 9525 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:22.033840 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.034406 9451 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
W20260503 13:58:22.010509 10245 log.cc:927] Time spent T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.145s user 0.000s sys 0.000s
I20260503 13:58:21.986243 9519 raft_consensus.cc:2393] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 2: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:21.978937 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.064216 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.010059 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.010385 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:21.923692 9600 rpcz_store.cc:276] 0503 13:58:19.810731 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.502625 (+1691894us) service_pool.cc:224] Handling call
0503 13:58:21.522547 (+ 19922us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:21.852642 (+330095us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.912799 (+ 60157us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x7b5400058a54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:21.912958 (+ 159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":669226880}
I20260503 13:58:22.126204 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.127787 9600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 215) took 2244 ms. Trace:
I20260503 13:58:22.124464 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.128091 9600 rpcz_store.cc:276] 0503 13:58:19.883544 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.127007 (+2243463us) service_pool.cc:224] Handling call
0503 13:58:22.127307 (+ 300us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.127474 (+ 167us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:22.127501 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:22.127546 (+ 45us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:22.127575 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.127634 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:22.127737 (+ 103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.129696 10200 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.131595 10200 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:21.877138 9447 rpcz_store.cc:276] 0503 13:58:17.956950 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.346280 (+3389330us) service_pool.cc:224] Handling call
0503 13:58:21.378356 (+ 32076us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b5800000998. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:21.378545 (+ 189us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:21.378862 (+ 317us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:21.378892 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:21.379351 (+ 459us) log.cc:844] Serialized 26 byte log entry
0503 13:58:21.500458 (+121107us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:21.500542 (+ 84us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.501194 (+ 652us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:21.808930 (+307736us) raft_consensus.cc:1685] finished
0503 13:58:21.811653 (+ 2723us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:21.831107 (+ 19454us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":966,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":16978,"spinlock_wait_cycles":33579776,"thread_start_us":120753,"threads_started":1,"wal-append.queue_time_us":143506}
I20260503 13:58:22.132988 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.134055 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.134359 10089 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.136757 10089 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:21.979625 10028 raft_consensus.cc:2804] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260503 13:58:22.119501 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 298315us
I20260503 13:58:21.983397 9444 raft_consensus.cc:1240] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.159425 10188 consensus_queue.cc:260] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.135111 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.139761 10129 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.163071 10129 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.141901 9644 tablet_bootstrap.cc:654] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
W20260503 13:58:22.142532 9417 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46726 dropped due to backpressure: service queue is full with 50 items
I20260503 13:58:22.033581 9445 raft_consensus.cc:2468] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:22.028728 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:21.919533 9450 rpcz_store.cc:276] 0503 13:58:20.600718 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.069264 (+468546us) service_pool.cc:224] Handling call
0503 13:58:21.873599 (+804335us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b54001a09d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:21.876040 (+ 2441us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1296374016}
W20260503 13:58:22.114830 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 137805us
I20260503 13:58:22.064949 10079 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.156937 9525 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:22.011133 10243 raft_consensus.cc:2749] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:22.158512 9416 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:22.031875 9596 raft_consensus.cc:2468] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:22.011261 10224 raft_consensus.cc:515] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:22.161475 10191 consensus_meta.cc:287] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.720s user 0.015s sys 0.059s
I20260503 13:58:21.904001 9452 rpcz_store.cc:276] 0503 13:58:18.156675 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.430231 (+3273556us) service_pool.cc:224] Handling call
0503 13:58:21.479200 (+ 48969us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:21.605954 (+126754us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:21.837124 (+231170us) spinlock_profiling.cc:243] Waited 31.9 ms on lock 0x7b54001d24d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:21.849731 (+ 12607us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":73426304}
I20260503 13:58:22.159163 10028 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:22.170630 9567 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:22.171500 9416 leader_election.cc:400] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Vote denied by peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997) with higher term. Message: Invalid argument: T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3.
I20260503 13:58:22.178329 9491 leader_election.cc:304] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:22.178625 9416 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.179992 9416 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.172344 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.149829 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:22.167873 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:22.164801 9898 consensus_queue.cc:1059] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.004s
I20260503 13:58:22.182307 9566 leader_election.cc:304] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:22.171880 9599 raft_consensus.cc:1240] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260503 13:58:22.184799 9416 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:22.181926 10028 raft_consensus.cc:3060] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:22.188818 10166 raft_consensus.cc:2804] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:22.189468 10166 raft_consensus.cc:697] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:22.190323 10166 consensus_queue.cc:237] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.174983 10079 raft_consensus.cc:2804] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:22.204680 10079 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:22.205065 10079 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:22.205349 10017 raft_consensus.cc:2749] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260503 13:58:22.205994 9565 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:22.206377 10253 raft_consensus.cc:2804] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:22.206763 10253 raft_consensus.cc:493] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:22.207051 10253 raft_consensus.cc:3060] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:22.195143 9416 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.195809 9898 consensus_queue.cc:1243] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer 4bea375a55034750a1f8804b0db97868 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260503 13:58:22.176309 10191 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.198503 10200 raft_consensus.cc:2804] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:22.176770 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.198256 10099 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.188930 10260 raft_consensus.cc:2749] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:22.199213 10201 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:22.217666 9449 raft_consensus.cc:2393] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.202922 10224 leader_election.cc:290] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:22.218482 10160 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.218809 9445 raft_consensus.cc:2393] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.182835 10240 raft_consensus.cc:2804] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:22.220939 10240 raft_consensus.cc:493] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:22.221274 10240 raft_consensus.cc:3060] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:22.193547 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.175127 10184 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.215435 10170 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:22.214213 9991 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:22.194950 10179 consensus_meta.cc:287] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.684s user 0.010s sys 0.007s
I20260503 13:58:22.220141 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.203814 9451 raft_consensus.cc:2468] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:22.183871 10129 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:22.194900 9600 raft_consensus.cc:1240] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:22.229028 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.229429 9991 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.229707 9521 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:22.217794 10201 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.219354 10200 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:22.233162 9596 raft_consensus.cc:1240] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:22.234763 9567 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:22.229036 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.242491 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.242789 9991 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.214363 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.243176 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.276024 9416 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.257707 9642 raft_consensus.cc:359] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.225077 10039 raft_consensus.cc:2749] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997) with higher term. Message: Invalid argument: T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4.
I20260503 13:58:22.286960 9642 raft_consensus.cc:385] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:22.286829 9418 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:22.287571 9642 raft_consensus.cc:740] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:22.287827 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.288352 9525 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:22.223838 10168 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.291254 9351 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51400 (request call id 69) took 4709 ms. Trace:
W20260503 13:58:22.292719 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 272969us
I20260503 13:58:22.243191 10200 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:22.243432 9974 raft_consensus.cc:2749] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:22.253073 9528 raft_consensus.cc:2393] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 2: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:22.291383 9642 consensus_queue.cc:260] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.294782 9601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 214) took 2443 ms. Trace:
I20260503 13:58:22.291613 9351 rpcz_store.cc:276] 0503 13:58:17.576910 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:17.577045 (+ 135us) service_pool.cc:224] Handling call
0503 13:58:22.286100 (+4709055us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0503 13:58:21.785159 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 13:58:21.785717 (+ 558us) write_op.cc:432] Acquiring schema lock in shared mode
0503 13:58:21.785765 (+ 48us) write_op.cc:435] Acquired schema lock
0503 13:58:21.785783 (+ 18us) tablet.cc:662] Decoding operations
0503 13:58:21.795520 (+ 9737us) write_op.cc:620] Acquiring the partition lock for write op
0503 13:58:21.795613 (+ 93us) write_op.cc:641] Partition lock acquired for write op
0503 13:58:21.795640 (+ 27us) tablet.cc:685] Acquiring locks for 16 operations
0503 13:58:21.796784 (+ 1144us) tablet.cc:701] Row locks acquired
0503 13:58:21.796804 (+ 20us) write_op.cc:260] PREPARE: finished
0503 13:58:21.796958 (+ 154us) write_op.cc:270] Start()
0503 13:58:21.797080 (+ 122us) write_op.cc:276] Timestamp: P: 1777816701796900 usec, L: 0
0503 13:58:21.797116 (+ 36us) op_driver.cc:348] REPLICATION: starting
0503 13:58:21.797761 (+ 645us) log.cc:844] Serialized 6313 byte log entry
0503 13:58:21.913121 (+115360us) op_driver.cc:464] REPLICATION: finished
0503 13:58:21.985651 (+ 72530us) write_op.cc:301] APPLY: starting
0503 13:58:22.009949 (+ 24298us) tablet.cc:1367] starting BulkCheckPresence
0503 13:58:22.010272 (+ 323us) tablet.cc:1370] finished BulkCheckPresence
0503 13:58:22.010390 (+ 118us) tablet.cc:1372] starting ApplyRowOperation cycle
0503 13:58:22.015692 (+ 5302us) tablet.cc:1383] finished ApplyRowOperation cycle
0503 13:58:22.015800 (+ 108us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b18001e6040. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea5bbe6 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f416d285c 00007f9f3ea70c0b
0503 13:58:22.015932 (+ 132us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0503 13:58:22.015982 (+ 50us) write_op.cc:312] APPLY: finished
0503 13:58:22.016336 (+ 354us) log.cc:844] Serialized 112 byte log entry
0503 13:58:22.017477 (+ 1141us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b18001e6040. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f416d285c 00007f9f3ea70c0b 00007f9f47869bc9 00007f9f3ea8ccdb 00007f9f3ea8af7f 00007f9f3ea8d242 00007f9f4556d360 00007f9f4556d8f9 00007f9f4556fb8a
0503 13:58:22.033516 (+ 16039us) write_op.cc:489] Releasing partition, row and schema locks
0503 13:58:22.158442 (+124926us) write_op.cc:454] Released schema lock
0503 13:58:22.168821 (+ 10379us) write_op.cc:341] FINISH: Updating metrics
0503 13:58:22.195221 (+ 26400us) write_op.cc:489] Releasing partition, row and schema locks
0503 13:58:22.209661 (+ 14440us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1545,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":9894,"apply.run_cpu_time_us":8592,"apply.run_wall_time_us":203238,"num_ops":16,"prepare.queue_time_us":85469,"prepare.run_cpu_time_us":5280,"prepare.run_wall_time_us":29672,"raft.queue_time_us":47748,"raft.run_cpu_time_us":7495,"raft.run_wall_time_us":104766,"replication_time_us":115849,"spinlock_wait_cycles":54210176,"thread_start_us":139752,"threads_started":4,"wal-append.queue_time_us":2091}]]}
I20260503 13:58:22.255016 10184 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:22.253166 10089 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:22.260133 10188 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:22.257848 10201 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.253991 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:22.253114 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:22.295387 10175 consensus_meta.cc:287] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.501s user 0.036s sys 0.037s
I20260503 13:58:22.289057 10260 raft_consensus.cc:2749] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:22.165302 9417 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46726 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 266)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 298)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 274)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 276)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 270)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 281)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 291)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 275)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 294)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 300)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 290)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 278)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 280)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 256)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 258)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 227)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 230)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 231)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 260)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 261)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 268)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 270)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 275)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 277)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 283)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 289)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 292)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 260)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 296)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 297)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 299)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 268)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 272)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 273)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 276)
I20260503 13:58:22.290529 9991 raft_consensus.cc:2804] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:22.297125 9601 rpcz_store.cc:276] 0503 13:58:19.851718 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.760823 (+1909105us) service_pool.cc:224] Handling call
0503 13:58:21.768986 (+ 8163us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:21.769191 (+ 205us) raft_consensus.cc:1534] Early marking committed up to index 1
0503 13:58:21.944773 (+175582us) log.cc:844] Serialized 14 byte log entry
0503 13:58:22.294426 (+349653us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:22.294471 (+ 45us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:22.294510 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.294635 (+ 125us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:22.294734 (+ 99us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":374268,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":1159,"thread_start_us":486747,"threads_started":2}
I20260503 13:58:22.245384 10202 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.285056 9527 raft_consensus.cc:2468] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:22.238759 10179 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.295909 10193 raft_consensus.cc:3060] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:22.303748 10191 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.314419 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:22.309414 9596 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:22.314999 9444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 254) took 4308 ms. Trace:
I20260503 13:58:22.315120 10239 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.315430 10171 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.288946 10159 consensus_queue.cc:1059] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260503 13:58:22.320137 9525 raft_consensus.cc:2468] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:22.320801 9642 ts_tablet_manager.cc:1434] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.385s user 0.007s sys 0.004s
I20260503 13:58:22.312523 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.323559 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:22.323931 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 218) took 2142 ms. Trace:
I20260503 13:58:22.324855 10251 raft_consensus.cc:3055] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:22.326828 10240 raft_consensus.cc:515] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.306875 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:22.329078 9418 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.317893 9991 raft_consensus.cc:697] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:22.332984 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.341298 9418 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.306973 10184 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.343650 9446 raft_consensus.cc:2393] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.313218 10175 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.343084 10239 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.330713 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.387887 9526 raft_consensus.cc:2368] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3.
I20260503 13:58:22.344348 9416 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:22.327440 10028 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.330049 9492 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.341670 9528 raft_consensus.cc:3060] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:22.370976 9991 consensus_queue.cc:237] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.371618 10184 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.377287 9642 tablet_bootstrap.cc:492] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:22.343454 10191 raft_consensus.cc:2804] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:22.371980 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.387382 9597 raft_consensus.cc:2393] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
W20260503 13:58:22.390946 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 60954.2us
I20260503 13:58:22.376757 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.340493 10251 raft_consensus.cc:740] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:22.351636 10240 leader_election.cc:290] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:22.345011 9449 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 251) took 1731 ms (client timeout 2232 ms). Trace:
I20260503 13:58:22.398363 9492 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:22.391072 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 46796.9us
I20260503 13:58:22.344929 9522 raft_consensus.cc:2468] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:22.381026 10260 raft_consensus.cc:2804] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:22.391736 10202 raft_consensus.cc:2804] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:22.391510 10099 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.399293 10191 raft_consensus.cc:697] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:22.391889 10159 raft_consensus.cc:2749] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260503 13:58:22.397782 10238 consensus_meta.cc:287] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.543s user 0.014s sys 0.002s
I20260503 13:58:22.420583 10251 consensus_queue.cc:260] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.420825 10238 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.421723 9453 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:22.422739 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.423807 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.406852 9416 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.407487 9524 raft_consensus.cc:2368] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3.
I20260503 13:58:22.407994 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.406406 9492 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.403664 9593 raft_consensus.cc:2468] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:22.413278 9567 leader_election.cc:400] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Vote denied by peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805) with higher term. Message: Invalid argument: T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3.
I20260503 13:58:22.412773 10202 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:22.412092 10129 raft_consensus.cc:2749] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:22.330431 9444 rpcz_store.cc:276] 0503 13:58:18.006155 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.371418 (+3365263us) service_pool.cc:224] Handling call
0503 13:58:21.371728 (+ 310us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.160129 (+788401us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.314790 (+154661us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b54002e68d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:22.314943 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1308726144}
I20260503 13:58:22.415256 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.415850 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:22.412092 10204 consensus_meta.cc:287] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.596s user 0.006s sys 0.007s
I20260503 13:58:22.340173 9598 rpcz_store.cc:276] 0503 13:58:20.181889 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.128887 (+1946998us) service_pool.cc:224] Handling call
0503 13:58:22.129172 (+ 285us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.129369 (+ 197us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:22.289889 (+160520us) log.cc:844] Serialized 14 byte log entry
0503 13:58:22.323612 (+ 33723us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:22.323667 (+ 55us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:22.323706 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.323770 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:22.323879 (+ 109us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14134,"prepare.queue_time_us":47034,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":746,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"thread_start_us":95283,"threads_started":2,"wal-append.queue_time_us":92741}
I20260503 13:58:22.418373 10028 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.419453 10171 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.424948 9567 leader_election.cc:403] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260503 13:58:22.423724 10238 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.419801 9528 raft_consensus.cc:2468] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:22.412879 9594 raft_consensus.cc:2393] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
W20260503 13:58:22.405547 9449 rpcz_store.cc:269] 0503 13:58:20.610271 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.068264 (+457993us) service_pool.cc:224] Handling call
0503 13:58:22.330928 (+1262664us) spinlock_profiling.cc:243] Waited 912 ms on lock 0x7b54002945d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:22.341222 (+ 10294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2097298176}
I20260503 13:58:22.437273 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.437518 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.437813 9594 raft_consensus.cc:3060] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:22.424530 9453 raft_consensus.cc:2393] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.438331 9449 raft_consensus.cc:2393] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.414508 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:22.420293 10260 raft_consensus.cc:697] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:22.420806 10191 consensus_queue.cc:237] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.402594 10170 raft_consensus.cc:2749] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:22.439033 9526 raft_consensus.cc:2468] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:22.421914 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:22.439486 10260 consensus_queue.cc:237] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.440039 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:22.440380 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.440594 9522 raft_consensus.cc:1240] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.440740 9526 raft_consensus.cc:2393] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 2: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:22.425057 10202 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:22.440977 9453 raft_consensus.cc:2468] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:22.441771 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.442261 9453 raft_consensus.cc:2393] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.442925 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.425275 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.426940 9418 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:22.449021 9416 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.449451 9418 leader_election.cc:304] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:22.403266 9519 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:22.447932 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 270) took 2533 ms. Trace:
I20260503 13:58:22.421761 10251 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:22.426450 9492 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.451618 9451 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:22.453455 9418 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:22.439092 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.454152 9528 rpcz_store.cc:276] 0503 13:58:19.914368 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.447212 (+2532844us) service_pool.cc:224] Handling call
0503 13:58:22.447469 (+ 257us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.447614 (+ 145us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:22.447629 (+ 15us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:22.447655 (+ 26us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:22.447679 (+ 24us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.447792 (+ 113us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:22.447885 (+ 93us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.446586 9520 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:22.428248 9523 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:22.428776 10200 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:22.434158 10241 consensus_meta.cc:287] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.532s user 0.004s sys 0.012s
I20260503 13:58:22.435067 10129 raft_consensus.cc:2749] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:22.434595 10253 raft_consensus.cc:515] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.454787 10170 raft_consensus.cc:2804] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:22.457336 9522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 268) took 2586 ms. Trace:
I20260503 13:58:22.432875 10175 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.456935 10028 raft_consensus.cc:2749] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:22.434898 10204 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.430034 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.452451 10238 raft_consensus.cc:2804] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:22.454792 10201 raft_consensus.cc:2749] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:22.426642 10159 raft_consensus.cc:2804] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:22.443342 10091 raft_consensus.cc:3060] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:22.447883 10039 raft_consensus.cc:2749] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:22.427145 9417 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:46726 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 290)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 278)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 280)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 284)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 288)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 291)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 256)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 258)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 227)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 230)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 231)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 260)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 261)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 268)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 270)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 275)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 277)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 283)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 289)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 292)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 260)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 296)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 297)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 299)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 268)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 272)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 273)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 276)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 285)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 289)
I20260503 13:58:22.461935 10184 raft_consensus.cc:2804] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:22.476311 9644 tablet_bootstrap.cc:492] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:22.432443 10193 raft_consensus.cc:2749] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:22.427701 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.428961 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.458413 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.539463 9528 raft_consensus.cc:2393] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 2: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:22.461040 10089 raft_consensus.cc:2749] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:22.534458 9643 tablet_bootstrap.cc:654] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:22.535655 10179 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:22.539897 9644 ts_tablet_manager.cc:1403] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.622s user 0.022s sys 0.038s
I20260503 13:58:22.539851 9492 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.542593 9492 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.478850 10170 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:22.545744 9492 leader_election.cc:304] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.547878 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:22.531594 10171 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:22.461782 10079 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:22.551867 9567 leader_election.cc:343] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:22.552832 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.535211 10238 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:22.554870 9528 raft_consensus.cc:2393] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:22.537743 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.553093 9416 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.479135 10241 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.547942 10170 raft_consensus.cc:3060] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:22.549283 9898 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:22.547209 10184 raft_consensus.cc:697] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:22.541872 10159 raft_consensus.cc:697] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:22.546862 9444 raft_consensus.cc:2468] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:22.593869 9596 raft_consensus.cc:2393] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 1: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:22.547135 10240 raft_consensus.cc:2749] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:22.554141 9521 raft_consensus.cc:2393] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:22.606496 9525 raft_consensus.cc:1240] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.555994 10238 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:22.606838 9644 raft_consensus.cc:359] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:22.607340 10199 consensus_meta.cc:287] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.813s user 0.003s sys 0.011s
I20260503 13:58:22.555168 10171 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.604908 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.604688 10159 consensus_queue.cc:237] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:22.605904 10268 log.cc:927] Time spent T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260503 13:58:22.478902 9522 rpcz_store.cc:276] 0503 13:58:19.870477 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.440111 (+2569634us) service_pool.cc:224] Handling call
0503 13:58:22.440392 (+ 281us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.450726 (+ 10334us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.457281 (+ 6555us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.603370 9528 raft_consensus.cc:1240] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.609272 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.609483 9492 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.641986 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 275) took 2079 ms. Trace:
I20260503 13:58:22.597481 10260 raft_consensus.cc:2749] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:22.610718 9521 raft_consensus.cc:1240] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:22.615701 9594 raft_consensus.cc:2468] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
W20260503 13:58:22.616859 10265 log.cc:927] Time spent T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.215s user 0.001s sys 0.000s
I20260503 13:58:22.633157 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.621043 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 213) took 2776 ms. Trace:
I20260503 13:58:22.621549 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:22.622833 10073 consensus_meta.cc:287] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.702s user 0.004s sys 0.013s
I20260503 13:58:22.617272 10160 raft_consensus.cc:3055] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:22.612524 9642 tablet_bootstrap.cc:654] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:22.595615 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.631036 9644 raft_consensus.cc:385] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:22.656138 9644 raft_consensus.cc:740] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:22.657828 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 271) took 2734 ms. Trace:
I20260503 13:58:22.631434 9418 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.657974 9644 consensus_queue.cc:260] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.632272 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.632550 10241 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.607689 10199 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.634457 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.596733 10099 raft_consensus.cc:493] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:22.654639 9599 rpcz_store.cc:276] 0503 13:58:19.845034 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.506481 (+1661447us) service_pool.cc:224] Handling call
0503 13:58:21.833890 (+327409us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b5800010898. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:21.849926 (+ 16036us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:22.193918 (+343992us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.620811 (+426893us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b540031b5d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:22.620990 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":675281920}
W20260503 13:58:22.665400 9597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 181) took 2104 ms (client timeout 2250 ms). Trace:
I20260503 13:58:22.665304 10099 raft_consensus.cc:515] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.666277 9448 raft_consensus.cc:2368] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3.
I20260503 13:58:22.666996 9524 raft_consensus.cc:1240] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:22.667230 10099 leader_election.cc:290] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.615640 9453 raft_consensus.cc:2393] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.668079 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.668234 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.639721 10238 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.641222 9566 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:22.641978 9451 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:22.644253 9492 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.642503 9528 rpcz_store.cc:276] 0503 13:58:20.562165 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.593738 (+2031573us) service_pool.cc:224] Handling call
0503 13:58:22.603250 (+ 9512us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.641805 (+ 38555us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.641946 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:22.638376 10196 consensus_meta.cc:287] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.601s user 0.010s sys 0.004s
I20260503 13:58:22.635802 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.629385 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:22.617251 9593 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 242) took 1342 ms (client timeout 1661 ms). Trace:
I20260503 13:58:22.637418 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.650463 10204 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:22.658455 9525 rpcz_store.cc:276] 0503 13:58:19.922991 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.551088 (+2628097us) service_pool.cc:224] Handling call
0503 13:58:22.598249 (+ 47161us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.606825 (+ 8576us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.657647 (+ 50822us) spinlock_profiling.cc:243] Waited 8.09 ms on lock 0x7b540004e054. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:22.657779 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18596224}
I20260503 13:58:22.659418 9491 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:22.695363 9451 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:22.659844 9523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 306) took 1149 ms. Trace:
I20260503 13:58:22.696523 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.657883 10073 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.661829 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.698024 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.698400 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.698693 9451 raft_consensus.cc:2393] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.640122 10202 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.699450 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.700095 9451 raft_consensus.cc:2393] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:22.700896 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.701047 10202 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.663590 9567 leader_election.cc:400] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805) with higher term. Message: Invalid argument: T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3.
I20260503 13:58:22.701934 9567 leader_election.cc:403] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260503 13:58:22.701879 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.702625 9451 raft_consensus.cc:2393] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.703332 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.663398 9416 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.704216 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.659430 10184 consensus_queue.cc:237] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.665681 10199 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.598773 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.633419 10079 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.676455 9600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 230) took 1652 ms. Trace:
W20260503 13:58:22.665812 9597 rpcz_store.cc:269] 0503 13:58:20.560902 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.288194 (+727292us) service_pool.cc:224] Handling call
0503 13:58:22.665232 (+1377038us) spinlock_profiling.cc:243] Waited 971 ms on lock 0x7b540031abd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:22.665347 (+ 115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2233981696}
I20260503 13:58:22.643375 10193 consensus_queue.cc:1059] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260503 13:58:22.673746 9528 raft_consensus.cc:1240] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.668928 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.662139 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.657019 10160 raft_consensus.cc:740] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:22.652825 10089 consensus_queue.cc:1059] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
W20260503 13:58:22.671762 9565 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:22.675938 10166 consensus_queue.cc:1059] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260503 13:58:22.677924 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.679097 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:22.722216 9601 raft_consensus.cc:2468] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:22.666201 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.677664 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.677584 10191 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:22.662495 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.662470 9521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 279) took 2093 ms. Trace:
I20260503 13:58:22.681824 9644 ts_tablet_manager.cc:1434] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.142s user 0.006s sys 0.000s
I20260503 13:58:22.681267 9452 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:22.678455 10196 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:22.681444 9593 rpcz_store.cc:269] 0503 13:58:21.274662 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.349261 (+ 74599us) service_pool.cc:224] Handling call
0503 13:58:22.617054 (+1267793us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540031b0d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:22.617192 (+ 138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2419862784}
I20260503 13:58:22.688972 10171 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.689253 9524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 204) took 1792 ms. Trace:
I20260503 13:58:22.730369 9524 rpcz_store.cc:276] 0503 13:58:20.896840 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.666442 (+1769602us) service_pool.cc:224] Handling call
0503 13:58:22.666770 (+ 328us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:22.674402 (+ 7632us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.689185 (+ 14783us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.731485 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.732198 9524 raft_consensus.cc:2393] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:22.733170 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:22.734272 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:22.735349 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.735680 10079 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:22.735993 9593 raft_consensus.cc:2393] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:22.736127 10079 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.737680 10196 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.738436 10079 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.712651 9600 rpcz_store.cc:276] 0503 13:58:21.022209 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.130601 (+1108392us) service_pool.cc:224] Handling call
0503 13:58:22.130867 (+ 266us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.244510 (+113643us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.662987 (+418477us) spinlock_profiling.cc:243] Waited 47.2 ms on lock 0x7b54000abc54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:22.674096 (+ 11109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":108497280}
I20260503 13:58:22.739161 9446 raft_consensus.cc:2393] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.739455 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:22.740414 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:22.741364 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.741858 9491 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:22.725340 9521 rpcz_store.cc:276] 0503 13:58:20.567505 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.607767 (+2040262us) service_pool.cc:224] Handling call
0503 13:58:22.610548 (+ 2781us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.651867 (+ 41319us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.660784 (+ 8917us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.742836 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.743319 9520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 281) took 2175 ms. Trace:
I20260503 13:58:22.690897 10091 raft_consensus.cc:2749] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805) with higher term. Message: Invalid argument: T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3.
I20260503 13:58:22.675936 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.744378 10191 raft_consensus.cc:697] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:22.745034 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.745227 10191 consensus_queue.cc:237] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.745823 10017 raft_consensus.cc:3055] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:22.671561 9519 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:22.746174 10017 raft_consensus.cc:740] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:22.746805 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.747313 9519 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:22.747745 10073 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.669384 10200 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.674260 10179 raft_consensus.cc:2749] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:22.697304 9523 rpcz_store.cc:276] 0503 13:58:21.510460 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:21.728954 (+218494us) service_pool.cc:224] Handling call
0503 13:58:22.659703 (+930749us) spinlock_profiling.cc:243] Waited 504 ms on lock 0x7b54002067d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:22.659795 (+ 92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1159668608}
I20260503 13:58:22.678707 10099 raft_consensus.cc:2749] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:22.727547 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.684358 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.748819 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.749217 9452 raft_consensus.cc:2393] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.749503 9522 raft_consensus.cc:2393] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 2: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:22.749871 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.750197 10200 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:22.750525 9522 raft_consensus.cc:1240] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:22.750634 10200 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.751124 9522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 283) took 1841 ms. Trace:
I20260503 13:58:22.751492 9522 rpcz_store.cc:276] 0503 13:58:20.909772 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.750061 (+1840289us) service_pool.cc:224] Handling call
0503 13:58:22.750347 (+ 286us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.750913 (+ 566us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.751076 (+ 163us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.688153 9898 raft_consensus.cc:2749] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:22.696858 9492 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.661016 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:22.752817 9445 raft_consensus.cc:2393] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.753719 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.753901 9595 raft_consensus.cc:1240] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:22.754557 9595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 193) took 1474 ms. Trace:
I20260503 13:58:22.754706 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.755312 9445 raft_consensus.cc:2468] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:22.754889 9595 rpcz_store.cc:276] 0503 13:58:21.280482 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.753447 (+1472965us) service_pool.cc:224] Handling call
0503 13:58:22.753727 (+ 280us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.754317 (+ 590us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.754504 (+ 187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.756106 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.704636 9416 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:22.757037 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 52499us
I20260503 13:58:22.757532 9567 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:22.757913 9642 tablet_bootstrap.cc:492] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:22.758111 9416 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.758337 9642 ts_tablet_manager.cc:1403] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.414s user 0.009s sys 0.003s
I20260503 13:58:22.664698 9449 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:22.657151 10028 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:22.742010 10224 consensus_queue.cc:1059] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20260503 13:58:22.742218 9453 raft_consensus.cc:2393] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.711342 10202 raft_consensus.cc:2749] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:22.713110 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 205) took 1811 ms. Trace:
I20260503 13:58:22.714701 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:22.714141 9350 catalog_manager.cc:5671] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 1 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:22.743542 9521 raft_consensus.cc:2393] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
W20260503 13:58:22.759564 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 62786.2us
I20260503 13:58:22.766592 9528 rpcz_store.cc:276] 0503 13:58:20.902073 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.673335 (+1771262us) service_pool.cc:224] Handling call
0503 13:58:22.673596 (+ 261us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.712918 (+ 39322us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.713066 (+ 148us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.704725 9974 raft_consensus.cc:2804] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:22.715409 10170 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.717234 10193 raft_consensus.cc:3055] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
W20260503 13:58:22.719570 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 47159.7us
I20260503 13:58:22.720827 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:22.710863 10248 raft_consensus.cc:3060] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 3
I20260503 13:58:22.761489 9445 raft_consensus.cc:3060] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:22.720676 9444 raft_consensus.cc:2393] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.769877 9974 raft_consensus.cc:697] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:22.770327 10204 raft_consensus.cc:3055] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:22.770567 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:22.768476 10091 consensus_queue.cc:1059] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260503 13:58:22.749614 9452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 286) took 1055 ms. Trace:
I20260503 13:58:22.771437 9452 rpcz_store.cc:276] 0503 13:58:21.694514 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.727296 (+1032782us) service_pool.cc:224] Handling call
0503 13:58:22.749562 (+ 22266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.771878 9444 raft_consensus.cc:1240] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.705029 10163 consensus_queue.cc:1059] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260503 13:58:22.772442 10028 raft_consensus.cc:697] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:22.758688 10179 consensus_queue.cc:1059] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260503 13:58:22.760957 10196 raft_consensus.cc:2749] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:22.729467 9450 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:22.765148 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:22.760779 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:22.769546 9451 raft_consensus.cc:2393] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:22.745384 9599 raft_consensus.cc:1240] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.760582 10188 raft_consensus.cc:2749] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:22.767401 9350 catalog_manager.cc:5671] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 1 to 2, leader changed from 1cb63886040047559d3893923cc9aaf6 (127.8.26.193) to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 2 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:22.743328 10039 raft_consensus.cc:2764] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260503 13:58:22.676054 10253 leader_election.cc:290] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:22.753535 10160 consensus_queue.cc:260] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.712074 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.759298 9418 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:22.671357 9448 raft_consensus.cc:3060] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:22.759040 9416 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.793394 9416 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:22.794162 9416 leader_election.cc:343] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:22.794710 9416 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.761010 9644 tablet_bootstrap.cc:492] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:22.761583 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:22.767328 9492 leader_election.cc:343] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:22.769505 9566 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:22.765051 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 287) took 1070 ms. Trace:
I20260503 13:58:22.712916 10113 consensus_queue.cc:1059] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260503 13:58:22.765182 9898 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:22.752468 10200 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:22.742926 9491 leader_election.cc:343] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:22.801888 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 58990.8us
I20260503 13:58:22.775818 9451 raft_consensus.cc:1240] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:22.803236 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 230) took 3926 ms. Trace:
I20260503 13:58:22.803558 9451 rpcz_store.cc:276] 0503 13:58:18.876890 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.775412 (+3898522us) service_pool.cc:224] Handling call
0503 13:58:22.775674 (+ 262us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.803021 (+ 27347us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.803190 (+ 169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.783628 9526 raft_consensus.cc:1240] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.780054 9350 catalog_manager.cc:5671] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 2 to 3, leader changed from 1cb63886040047559d3893923cc9aaf6 (127.8.26.193) to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 3 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:22.727384 10238 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.789135 10079 consensus_queue.cc:1059] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260503 13:58:22.780088 10089 raft_consensus.cc:3055] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:22.795712 9446 rpcz_store.cc:276] 0503 13:58:21.694757 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.607228 (+912471us) service_pool.cc:224] Handling call
0503 13:58:22.764887 (+157659us) spinlock_profiling.cc:243] Waited 44.8 ms on lock 0x7b540017a054. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:22.765002 (+ 115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103084416}
I20260503 13:58:22.797994 10160 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:22.780303 10091 raft_consensus.cc:3055] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:22.772830 10163 raft_consensus.cc:3055] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:22.768481 10166 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:22.798580 10113 raft_consensus.cc:3055] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:22.799472 9566 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:22.799942 9978 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:22.772540 9452 raft_consensus.cc:1240] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:22.776079 9597 raft_consensus.cc:1275] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:22.801018 9898 raft_consensus.cc:697] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:22.805271 9350 catalog_manager.cc:5671] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 1 to 2, leader changed from 1cb63886040047559d3893923cc9aaf6 (127.8.26.193) to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 2 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:22.796051 9492 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:22.824010 9492 leader_election.cc:343] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:22.806753 10089 raft_consensus.cc:740] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:22.824518 9492 leader_election.cc:304] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.807521 9643 tablet_bootstrap.cc:492] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:22.761567 10073 raft_consensus.cc:2749] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:22.778749 9445 raft_consensus.cc:2468] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:22.810683 9525 raft_consensus.cc:1240] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.774632 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:22.784603 10193 raft_consensus.cc:740] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:22.781526 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.811652 10163 raft_consensus.cc:740] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:22.825263 9350 catalog_manager.cc:5671] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 2 to 3, leader changed from 1cb63886040047559d3893923cc9aaf6 (127.8.26.193) to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 3 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:22.772451 9444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 258) took 4458 ms. Trace:
I20260503 13:58:22.782585 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 191) took 1508 ms. Trace:
I20260503 13:58:22.776531 10204 raft_consensus.cc:740] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:22.776485 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.834221 9524 raft_consensus.cc:2393] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 2: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:22.833958 10089 consensus_queue.cc:260] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.834132 9898 consensus_queue.cc:237] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.835045 10089 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:22.835253 9524 raft_consensus.cc:1240] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.835836 9524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 290) took 1923 ms. Trace:
I20260503 13:58:22.826114 10166 raft_consensus.cc:697] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:22.778086 10170 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.826593 10188 raft_consensus.cc:2749] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:22.837028 10166 consensus_queue.cc:237] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.829452 10039 raft_consensus.cc:2749] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:22.837615 10163 consensus_queue.cc:260] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.838275 10238 raft_consensus.cc:3055] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:22.782276 9521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 287) took 1871 ms. Trace:
I20260503 13:58:22.831058 9453 raft_consensus.cc:1240] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.814814 9451 raft_consensus.cc:1240] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.839452 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 231) took 3848 ms. Trace:
I20260503 13:58:22.812952 10159 consensus_queue.cc:1059] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
W20260503 13:58:22.817457 9491 consensus_peers.cc:597] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a -> Peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Couldn't send request to peer 4bea375a55034750a1f8804b0db97868. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260503 13:58:22.801815 10199 raft_consensus.cc:2749] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:22.795909 10202 raft_consensus.cc:2804] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:22.807017 10196 raft_consensus.cc:2804] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:22.821768 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 269) took 2936 ms. Trace:
I20260503 13:58:22.822100 9601 raft_consensus.cc:1275] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:22.827553 10079 raft_consensus.cc:493] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:22.798280 9449 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:22.825796 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 288) took 1914 ms. Trace:
I20260503 13:58:22.822631 9598 raft_consensus.cc:2468] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:22.845378 9525 rpcz_store.cc:276] 0503 13:58:20.911513 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.790823 (+1879310us) service_pool.cc:224] Handling call
0503 13:58:22.810517 (+ 19694us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.825577 (+ 15060us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.825747 (+ 170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.808985 10091 raft_consensus.cc:740] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:22.826577 9643 ts_tablet_manager.cc:1403] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.953s user 0.015s sys 0.023s
I20260503 13:58:22.828744 9492 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.828650 9350 catalog_manager.cc:5671] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 reported cstate change: term changed from 2 to 3, leader changed from 1cb63886040047559d3893923cc9aaf6 (127.8.26.193) to 4bea375a55034750a1f8804b0db97868 (127.8.26.195). New cstate: current_term: 3 leader_uuid: "4bea375a55034750a1f8804b0db97868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:22.815929 10248 raft_consensus.cc:2749] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805) with higher term. Message: Invalid argument: T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3.
I20260503 13:58:22.816041 10113 raft_consensus.cc:740] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:22.831276 9446 raft_consensus.cc:1240] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.770732 9974 consensus_queue.cc:237] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.831554 9452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 227) took 4011 ms. Trace:
I20260503 13:58:22.818132 9991 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:22.832122 9593 raft_consensus.cc:1275] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:22.822988 10253 raft_consensus.cc:2749] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260503 13:58:22.832469 10204 consensus_queue.cc:260] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.882386 9991 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.883306 10204 raft_consensus.cc:3060] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:22.832988 10193 consensus_queue.cc:260] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.833312 9450 raft_consensus.cc:2468] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:22.833494 9596 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:22.773662 10028 consensus_queue.cc:237] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.813735 9416 leader_election.cc:304] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.818282 9566 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:22.839730 9451 rpcz_store.cc:276] 0503 13:58:18.990778 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.804267 (+3813489us) service_pool.cc:224] Handling call
0503 13:58:22.814661 (+ 10394us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.839255 (+ 24594us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.839407 (+ 152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.809415 10260 consensus_queue.cc:1059] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260503 13:58:22.842661 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:22.844122 9526 rpcz_store.cc:276] 0503 13:58:19.885529 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.441288 (+2555759us) service_pool.cc:224] Handling call
0503 13:58:22.441555 (+ 267us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.804417 (+362862us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.821571 (+ 17154us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b54000c62d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:22.821717 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":786305280}
W20260503 13:58:22.889158 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 70948us
I20260503 13:58:22.849577 10202 raft_consensus.cc:697] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:22.829790 9444 rpcz_store.cc:276] 0503 13:58:18.314414 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.771445 (+4457031us) service_pool.cc:224] Handling call
0503 13:58:22.771714 (+ 269us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.772226 (+ 512us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.772391 (+ 165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.847126 10163 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:22.836140 9524 rpcz_store.cc:276] 0503 13:58:20.911951 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.834799 (+1922848us) service_pool.cc:224] Handling call
0503 13:58:22.835099 (+ 300us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.835629 (+ 530us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.835789 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.916386 9978 raft_consensus.cc:2749] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:22.921468 10253 consensus_queue.cc:1059] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.009s
I20260503 13:58:22.930276 9598 raft_consensus.cc:1240] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:22.916992 10248 raft_consensus.cc:2749] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:22.928261 9519 raft_consensus.cc:2468] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:22.849025 10079 raft_consensus.cc:515] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.931447 9522 raft_consensus.cc:1240] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:22.848968 10243 consensus_queue.cc:1059] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:22.931869 9519 raft_consensus.cc:1240] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.932114 10243 raft_consensus.cc:3055] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:22.849660 9525 raft_consensus.cc:1240] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:22.932471 10243 raft_consensus.cc:740] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:22.932788 9522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 286) took 2021 ms. Trace:
I20260503 13:58:22.930500 9594 raft_consensus.cc:1240] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:22.932479 9519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 296) took 1918 ms. Trace:
I20260503 13:58:22.933451 9519 rpcz_store.cc:276] 0503 13:58:21.014009 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.931449 (+1917440us) service_pool.cc:224] Handling call
0503 13:58:22.931717 (+ 268us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.932242 (+ 525us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.932420 (+ 178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.866555 9491 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:22.843899 10196 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:22.830387 10017 consensus_queue.cc:260] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:22.879276 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 50593.1us
I20260503 13:58:22.935109 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 192) took 1660 ms. Trace:
I20260503 13:58:22.881577 9445 raft_consensus.cc:1240] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.881918 10113 consensus_queue.cc:260] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.932912 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 291) took 1979 ms. Trace:
I20260503 13:58:22.930620 10253 consensus_queue.cc:1243] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer 428b342dc4f041df956015e7d7cd173a log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260503 13:58:22.843969 10188 consensus_queue.cc:1048] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260503 13:58:22.936534 9525 rpcz_store.cc:276] 0503 13:58:20.953131 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.846061 (+1892930us) service_pool.cc:224] Handling call
0503 13:58:22.849519 (+ 3458us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.932710 (+ 83191us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.932866 (+ 156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.889081 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 259) took 4574 ms. Trace:
I20260503 13:58:22.937316 9978 raft_consensus.cc:3055] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:22.937721 10188 raft_consensus.cc:2804] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:22.935578 9492 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:22.937973 9978 raft_consensus.cc:740] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:22.937609 9453 rpcz_store.cc:276] 0503 13:58:18.314688 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.771931 (+4457243us) service_pool.cc:224] Handling call
0503 13:58:22.772209 (+ 278us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.840223 (+ 68014us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.880028 (+ 39805us) spinlock_profiling.cc:243] Waited 58.7 ms on lock 0x7b540029f754. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:22.887942 (+ 7914us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":134929152}
I20260503 13:58:22.936769 10113 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:22.939283 9453 raft_consensus.cc:1240] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.939802 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 271) took 2247 ms. Trace:
I20260503 13:58:22.939533 9978 consensus_queue.cc:260] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.940163 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 261) took 3174 ms. Trace:
I20260503 13:58:22.940681 9445 rpcz_store.cc:276] 0503 13:58:19.781132 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.850062 (+3068930us) service_pool.cc:224] Handling call
0503 13:58:22.878971 (+ 28909us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.939920 (+ 60949us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.940108 (+ 188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.941802 9445 raft_consensus.cc:1240] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.942365 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 275) took 1501 ms. Trace:
I20260503 13:58:22.942904 9445 rpcz_store.cc:276] 0503 13:58:21.441440 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.941343 (+1499903us) service_pool.cc:224] Handling call
0503 13:58:22.941643 (+ 300us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.942159 (+ 516us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.942319 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.943933 9445 raft_consensus.cc:1240] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.944420 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 277) took 1487 ms. Trace:
I20260503 13:58:22.944713 9445 rpcz_store.cc:276] 0503 13:58:21.490105 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.943526 (+1453421us) service_pool.cc:224] Handling call
0503 13:58:22.943794 (+ 268us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.944233 (+ 439us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.944378 (+ 145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.945757 10202 consensus_queue.cc:237] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.948933 10188 raft_consensus.cc:697] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
W20260503 13:58:22.941754 10275 log.cc:927] Time spent T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20260503 13:58:22.949733 10188 consensus_queue.cc:237] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.933251 10243 consensus_queue.cc:260] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.955192 10243 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:22.880869 9452 rpcz_store.cc:276] 0503 13:58:18.820090 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.772117 (+3952027us) service_pool.cc:224] Handling call
0503 13:58:22.772376 (+ 259us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.822715 (+ 50339us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.831498 (+ 8783us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.838809 9521 rpcz_store.cc:276] 0503 13:58:20.911289 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.781495 (+1870206us) service_pool.cc:224] Handling call
0503 13:58:22.781806 (+ 311us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.781975 (+ 169us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:22.782001 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:22.782044 (+ 43us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:22.782078 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.782133 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:22.782226 (+ 93us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:22.888515 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 74825.4us
I20260503 13:58:22.839565 10238 raft_consensus.cc:740] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:22.889416 9528 raft_consensus.cc:3060] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:22.887720 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 260) took 3145 ms. Trace:
I20260503 13:58:22.879398 9523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 284) took 1965 ms. Trace:
I20260503 13:58:22.917078 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:22.881299 10159 raft_consensus.cc:2749] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:22.922175 9566 leader_election.cc:400] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Vote denied by peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599) with higher term. Message: Invalid argument: T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3.
I20260503 13:58:22.849218 10091 consensus_queue.cc:260] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.888029 10200 raft_consensus.cc:2749] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:22.891415 10199 raft_consensus.cc:2749] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:22.916092 10193 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:22.930696 9642 raft_consensus.cc:359] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:22.999689 9642 raft_consensus.cc:385] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:22.857622 9449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 290) took 1154 ms. Trace:
I20260503 13:58:22.868620 9601 raft_consensus.cc:1240] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:22.836824 9599 rpcz_store.cc:276] 0503 13:58:21.274350 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.744915 (+1470565us) service_pool.cc:224] Handling call
0503 13:58:22.745208 (+ 293us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.782365 (+ 37157us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.782537 (+ 172us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.935885 9524 raft_consensus.cc:1240] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:22.884275 9991 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:22.935132 10196 raft_consensus.cc:3060] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:22.940097 9453 rpcz_store.cc:276] 0503 13:58:20.710728 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.938869 (+2228141us) service_pool.cc:224] Handling call
0503 13:58:22.939127 (+ 258us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.939609 (+ 482us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.939756 (+ 147us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:23.006572 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 47288.2us
I20260503 13:58:23.007861 9416 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:23.008200 9418 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:22.945703 9445 raft_consensus.cc:1240] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:22.979238 9492 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:22.978811 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 112349us
I20260503 13:58:22.937860 9525 raft_consensus.cc:1240] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:22.743782 9520 rpcz_store.cc:276] 0503 13:58:20.567981 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.618598 (+2050617us) service_pool.cc:224] Handling call
0503 13:58:22.618870 (+ 272us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.619051 (+ 181us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:22.660415 (+ 41364us) log.cc:844] Serialized 14 byte log entry
0503 13:58:22.742821 (+ 82406us) spinlock_profiling.cc:243] Waited 8.17 ms on lock 0x7b1800074ec0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f416d285c 00007f9f3ea70c0b 00007f9f47869bc9 00007f9f3ea8ccdb 00007f9f3ea8af7f 00007f9f3ea8d242 00007f9f416c45ab 00007f9f416ca0c3 00007f9f416ca5da
0503 13:58:22.742995 (+ 174us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:22.743046 (+ 51us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:22.743087 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.743148 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:22.743268 (+ 120us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":94216,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":1155,"raft.queue_time_us":5007,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"spinlock_wait_cycles":23951360,"thread_start_us":76101,"threads_started":2,"wal-append.queue_time_us":57199}
I20260503 13:58:22.982308 9526 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:23.019673 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 216) took 1765 ms. Trace:
I20260503 13:58:22.921613 10099 raft_consensus.cc:3055] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:22.989604 9978 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:22.999965 9642 raft_consensus.cc:740] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:22.933173 9522 rpcz_store.cc:276] 0503 13:58:20.911063 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.752232 (+1841169us) service_pool.cc:224] Handling call
0503 13:58:22.931265 (+179033us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.931804 (+ 539us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.932621 (+ 817us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b540013f1d0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:22.932738 (+ 117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":411060608}
I20260503 13:58:23.002031 9452 raft_consensus.cc:1240] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:23.002589 9444 raft_consensus.cc:1240] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
W20260503 13:58:23.002887 9494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:23.004176 9643 raft_consensus.cc:359] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:23.008327 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 86240us
I20260503 13:58:23.008992 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 283) took 1322 ms. Trace:
I20260503 13:58:23.005024 9449 rpcz_store.cc:276] 0503 13:58:21.695567 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.634185 (+938618us) service_pool.cc:224] Handling call
0503 13:58:22.850152 (+215967us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:23.025871 10171 raft_consensus.cc:2749] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:23.026059 9444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 289) took 1330 ms. Trace:
I20260503 13:58:22.980434 9446 rpcz_store.cc:276] 0503 13:58:19.750722 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.821576 (+3070854us) service_pool.cc:224] Handling call
0503 13:58:22.831118 (+ 9542us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.881271 (+ 50153us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.887674 (+ 6403us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:23.026542 9444 rpcz_store.cc:276] 0503 13:58:21.695271 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.002133 (+1306862us) service_pool.cc:224] Handling call
0503 13:58:23.002430 (+ 297us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.025840 (+ 23410us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.026010 (+ 170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:23.027541 9446 raft_consensus.cc:1240] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:23.028110 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 264) took 1225 ms. Trace:
I20260503 13:58:23.028417 9446 rpcz_store.cc:276] 0503 13:58:21.802360 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.027077 (+1224717us) service_pool.cc:224] Handling call
0503 13:58:23.027376 (+ 299us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.027889 (+ 513us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.028063 (+ 174us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:23.011904 10276 log.cc:927] Time spent T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260503 13:58:23.012023 9521 raft_consensus.cc:1240] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:23.013092 10224 consensus_queue.cc:1059] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:23.013799 10253 consensus_queue.cc:1059] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:23.014379 10204 consensus_queue.cc:1059] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:58:23.014923 10159 consensus_queue.cc:1048] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:23.021401 10099 raft_consensus.cc:740] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:23.015038 9416 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:23.015714 10260 raft_consensus.cc:2749] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:23.015519 10238 consensus_queue.cc:260] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.015038 9448 raft_consensus.cc:2468] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:23.017271 9453 raft_consensus.cc:1240] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:23.046952 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 292) took 1350 ms. Trace:
I20260503 13:58:23.047372 9453 rpcz_store.cc:276] 0503 13:58:21.696132 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.016874 (+1320742us) service_pool.cc:224] Handling call
0503 13:58:23.017132 (+ 258us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.046714 (+ 29582us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.046899 (+ 185us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:23.018261 9528 raft_consensus.cc:2468] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:23.019631 9418 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:23.020227 9525 rpcz_store.cc:276] 0503 13:58:21.254575 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.937364 (+1682789us) service_pool.cc:224] Handling call
0503 13:58:22.937652 (+ 288us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:23.009142 (+ 71490us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.019623 (+ 10481us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:22.987185 9595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 243) took 1154 ms. Trace:
I20260503 13:58:23.021665 9644 tablet_bootstrap.cc:654] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:22.935487 9594 rpcz_store.cc:276] 0503 13:58:21.274495 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.745879 (+1471384us) service_pool.cc:224] Handling call
0503 13:58:22.890359 (+144480us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:22.934256 (+ 43897us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.934935 (+ 679us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54003f07d0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:22.935047 (+ 112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":329904000}
I20260503 13:58:23.023032 9642 consensus_queue.cc:260] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.023720 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:23.030671 9524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 292) took 2067 ms. Trace:
I20260503 13:58:23.004671 10017 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:23.017351 9447 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:23.024744 9643 raft_consensus.cc:385] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:23.024899 9445 rpcz_store.cc:276] 0503 13:58:21.693594 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.945302 (+1251708us) service_pool.cc:224] Handling call
0503 13:58:22.945562 (+ 260us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.008784 (+ 63222us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.008947 (+ 163us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:23.025098 9566 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:23.061617 9643 raft_consensus.cc:740] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:23.005288 10079 leader_election.cc:290] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:23.062541 9451 raft_consensus.cc:1240] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:23.032378 9991 raft_consensus.cc:2804] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:23.025056 9491 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:23.065457 9526 raft_consensus.cc:1240] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:23.035259 9520 raft_consensus.cc:1240] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:23.044947 9416 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:23.068078 9416 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:23.048724 9453 raft_consensus.cc:1240] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:23.050740 10099 consensus_queue.cc:260] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.051211 10091 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:23.051647 10073 raft_consensus.cc:2749] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:23.046805 9452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 288) took 1346 ms. Trace:
I20260503 13:58:23.055598 9418 leader_election.cc:304] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:23.073921 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 49382us
I20260503 13:58:23.055831 9528 raft_consensus.cc:1240] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
W20260503 13:58:23.063354 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 49497.6us
I20260503 13:58:23.062279 9643 consensus_queue.cc:260] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.063556 9566 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:23.031800 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 52646.2us
I20260503 13:58:23.046934 10170 raft_consensus.cc:3055] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:23.070868 9526 raft_consensus.cc:1275] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:58:23.071578 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 299) took 1214 ms. Trace:
I20260503 13:58:23.072361 9525 raft_consensus.cc:1275] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:23.078553 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 268) took 3247 ms. Trace:
I20260503 13:58:23.076473 9492 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:23.078893 9451 rpcz_store.cc:276] 0503 13:58:19.831636 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.888963 (+3057327us) service_pool.cc:224] Handling call
0503 13:58:23.062395 (+173432us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.062884 (+ 489us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.078414 (+ 15530us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54003cfad0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:23.078508 (+ 94us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":337166720}
I20260503 13:58:23.062692 9978 raft_consensus.cc:3055] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:23.061992 9521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 304) took 1544 ms. Trace:
I20260503 13:58:23.063944 10191 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:23.078719 9453 rpcz_store.cc:276] 0503 13:58:21.856968 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.048151 (+1191183us) service_pool.cc:224] Handling call
0503 13:58:23.048423 (+ 272us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.071363 (+ 22940us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.071532 (+ 169us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:23.062243 10199 consensus_queue.cc:1175] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4bea375a55034750a1f8804b0db97868" responder_term: 4 status { last_received { term: 4 index: 3 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260503 13:58:23.085371 9526 raft_consensus.cc:1275] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:23.094450 9526 raft_consensus.cc:1275] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:23.094561 9445 raft_consensus.cc:1240] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:23.095083 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 268) took 1179 ms. Trace:
I20260503 13:58:23.095635 9526 raft_consensus.cc:1275] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260503 13:58:23.096727 9642 ts_tablet_manager.cc:1434] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.338s user 0.007s sys 0.006s
I20260503 13:58:23.095635 9445 rpcz_store.cc:276] 0503 13:58:21.915470 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.066577 (+1151107us) service_pool.cc:224] Handling call
0503 13:58:23.092098 (+ 25521us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b5800000998. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:23.094287 (+ 2189us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b5800000998. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea509ec 00007f9f3ea505ab 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:23.094416 (+ 129us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.094900 (+ 484us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.095036 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9437696}
I20260503 13:58:23.087172 9528 raft_consensus.cc:1275] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:23.072963 10238 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:23.084278 9492 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:23.089577 9991 raft_consensus.cc:697] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
W20260503 13:58:23.089897 9566 leader_election.cc:343] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:23.079968 9898 raft_consensus.cc:2804] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:23.120857 9566 leader_election.cc:304] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:23.078083 10170 raft_consensus.cc:740] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:23.073242 10224 raft_consensus.cc:2749] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:23.080102 10184 raft_consensus.cc:2749] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:23.127004 9642 tablet_bootstrap.cc:492] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:23.101025 9525 raft_consensus.cc:1275] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:23.063843 10260 consensus_queue.cc:1059] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260503 13:58:23.077939 10079 raft_consensus.cc:2749] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:23.121460 9492 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:23.086956 10191 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:23.055053 9524 rpcz_store.cc:276] 0503 13:58:20.957369 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.929659 (+1972290us) service_pool.cc:224] Handling call
0503 13:58:22.935717 (+ 6058us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.006462 (+ 70745us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.024371 (+ 17909us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:23.082559 9978 raft_consensus.cc:740] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:23.129194 9445 raft_consensus.cc:1240] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:23.087870 9520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 305) took 1575 ms. Trace:
I20260503 13:58:23.092665 10099 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:23.084766 10241 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:23.086740 10073 consensus_queue.cc:1048] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260503 13:58:23.133710 9451 raft_consensus.cc:1240] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260503 13:58:23.136463 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 276) took 1154 ms. Trace:
I20260503 13:58:23.077279 9452 rpcz_store.cc:276] 0503 13:58:21.695009 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.981178 (+1286169us) service_pool.cc:224] Handling call
0503 13:58:22.998906 (+ 17728us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.030117 (+ 31211us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.041541 (+ 11424us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:23.086053 10171 consensus_queue.cc:1059] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260503 13:58:23.087137 10251 consensus_queue.cc:1059] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260503 13:58:23.124321 10200 raft_consensus.cc:2804] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
W20260503 13:58:23.229565 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 92615us
I20260503 13:58:23.237581 9453 rpcz_store.cc:276] 0503 13:58:21.982561 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.087221 (+1104660us) service_pool.cc:224] Handling call
0503 13:58:23.135839 (+ 48618us) spinlock_profiling.cc:243] Waited 6.1 ms on lock 0x7b5800000998. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:23.136012 (+ 173us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.136178 (+ 166us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:23.136202 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:23.136241 (+ 39us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:23.136268 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.136319 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:23.136409 (+ 90us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14041472}
I20260503 13:58:23.139621 10191 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
W20260503 13:58:23.237818 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 147939us
W20260503 13:58:23.232538 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 143487us
I20260503 13:58:23.126053 10188 raft_consensus.cc:2749] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:23.129546 9522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 311) took 1279 ms. Trace:
I20260503 13:58:23.257968 9453 raft_consensus.cc:1275] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:23.267246 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 273) took 1332 ms. Trace:
I20260503 13:58:23.268352 9566 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:23.231666 10240 raft_consensus.cc:493] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:23.085080 9521 rpcz_store.cc:276] 0503 13:58:21.510159 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.999145 (+1488986us) service_pool.cc:224] Handling call
0503 13:58:23.011848 (+ 12703us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.033312 (+ 21464us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.054941 (+ 21629us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:23.237392 10159 raft_consensus.cc:2749] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:23.240724 9447 raft_consensus.cc:1275] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:23.267583 9451 rpcz_store.cc:276] 0503 13:58:21.935105 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.079753 (+1144648us) service_pool.cc:224] Handling call
0503 13:58:23.093957 (+ 14204us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:23.240496 (+146539us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.267045 (+ 26549us) spinlock_profiling.cc:243] Waited 39.5 ms on lock 0x7b540039d5d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:23.267200 (+ 155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":90882816}
I20260503 13:58:23.237628 10039 raft_consensus.cc:2804] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:23.249815 9519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 298) took 2235 ms. Trace:
W20260503 13:58:23.300411 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:23.269579 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 271) took 1343 ms. Trace:
W20260503 13:58:23.338219 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 62224.7us
I20260503 13:58:23.255475 10079 consensus_queue.cc:1059] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260503 13:58:23.273790 10200 raft_consensus.cc:697] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
W20260503 13:58:23.362236 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 240821us
I20260503 13:58:23.380345 9452 raft_consensus.cc:1240] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:23.274816 10170 consensus_queue.cc:260] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.307669 10188 raft_consensus.cc:3055] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:23.381194 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:23.381321 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:23.383514 10291 raft_consensus.cc:493] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:23.385625 10281 log.cc:927] Time spent T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.264s user 0.000s sys 0.000s
I20260503 13:58:23.268746 10292 consensus_queue.cc:1059] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.129s
I20260503 13:58:23.386448 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:23.387120 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:23.388866 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:23.406150 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 272) took 1441 ms. Trace:
W20260503 13:58:23.394745 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260503 13:58:23.551074 9416 leader_election.cc:343] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:23.395802 10196 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.295655 10171 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:23.274734 9991 consensus_queue.cc:237] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.412055 9492 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:23.221724 9520 rpcz_store.cc:276] 0503 13:58:21.510306 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.030574 (+1520268us) service_pool.cc:224] Handling call
0503 13:58:23.035090 (+ 4516us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.069414 (+ 34324us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.086135 (+ 16721us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:23.279456 10289 raft_consensus.cc:2749] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260503 13:58:23.433842 9491 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:22.981616 9523 rpcz_store.cc:276] 0503 13:58:20.909985 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.750677 (+1840692us) service_pool.cc:224] Handling call
0503 13:58:22.750957 (+ 280us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.751139 (+ 182us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:22.790755 (+ 39616us) log.cc:844] Serialized 14 byte log entry
0503 13:58:22.842883 (+ 52128us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:22.845327 (+ 2444us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:22.848741 (+ 3414us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.867042 (+ 18301us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:22.875192 (+ 8150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":16576,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":77,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":2555,"thread_start_us":30939,"threads_started":1,"wal-append.queue_time_us":92221}
I20260503 13:58:23.278463 10166 consensus_queue.cc:1059] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.143s
I20260503 13:58:23.337924 10296 raft_consensus.cc:493] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:23.583137 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:23.583870 9601 raft_consensus.cc:2393] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:23.367486 10283 consensus_queue.cc:1048] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.191s
W20260503 13:58:23.595057 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 326767us
I20260503 13:58:23.596951 9416 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:23.552141 10079 raft_consensus.cc:493] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:23.053131 9595 rpcz_store.cc:276] 0503 13:58:21.833149 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.756244 (+923095us) service_pool.cc:224] Handling call
0503 13:58:22.756553 (+ 309us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:22.812774 (+ 56221us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:22.813377 (+ 603us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:22.844495 (+ 31118us) log.cc:844] Serialized 26 byte log entry
0503 13:58:22.985242 (+140747us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:22.986218 (+ 976us) log.cc:844] Serialized 14 byte log entry
0503 13:58:22.986603 (+ 385us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:22.986655 (+ 52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:22.987035 (+ 380us) raft_consensus.cc:1685] finished
0503 13:58:22.987044 (+ 9us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:22.987139 (+ 95us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5249,"prepare.queue_time_us":962,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":7411,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":5254,"spinlock_wait_cycles":1956096,"thread_start_us":41209,"threads_started":2,"wal-append.queue_time_us":115155}
I20260503 13:58:23.479322 10193 consensus_queue.cc:1048] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.107s
I20260503 13:58:23.532430 9898 raft_consensus.cc:697] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
W20260503 13:58:23.533738 9974 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.002s sys 0.007s
I20260503 13:58:23.534827 9643 ts_tablet_manager.cc:1434] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.656s user 0.005s sys 0.008s
W20260503 13:58:23.674276 9643 ts_tablet_manager.cc:1467] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868: Tablet startup took 795ms
W20260503 13:58:23.535244 9565 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:23.535864 9450 raft_consensus.cc:1240] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:23.536269 10202 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:23.339103 9448 rpcz_store.cc:276] 0503 13:58:21.921725 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.073292 (+1151567us) service_pool.cc:224] Handling call
0503 13:58:23.098482 (+ 25190us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.231153 (+132671us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.264639 (+ 33486us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1408}
W20260503 13:58:23.399730 10160 consensus_meta.cc:287] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.588s user 0.009s sys 0.008s
I20260503 13:58:23.431135 10188 raft_consensus.cc:740] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:23.529844 10251 raft_consensus.cc:3055] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:23.277253 10240 raft_consensus.cc:515] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.270552 9522 rpcz_store.cc:276] 0503 13:58:21.850359 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.036635 (+1186276us) service_pool.cc:224] Handling call
0503 13:58:23.067765 (+ 31130us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.081031 (+ 13266us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:23.083422 (+ 2391us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:23.086115 (+ 2693us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:23.092815 (+ 6700us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.099856 (+ 7041us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:23.125833 (+ 25977us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:23.361713 10224 raft_consensus.cc:2749] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:23.447118 10184 consensus_queue.cc:1059] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.145s
I20260503 13:58:23.608690 10289 consensus_queue.cc:1048] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:23.580423 9598 raft_consensus.cc:2393] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
W20260503 13:58:23.598493 10163 consensus_meta.cc:287] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.668s user 0.009s sys 0.000s
W20260503 13:58:23.652833 10204 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.004s sys 0.007s
I20260503 13:58:23.673048 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:23.610250 10287 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:23.551312 9447 rpcz_store.cc:276] 0503 13:58:21.925590 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.079517 (+1153927us) service_pool.cc:224] Handling call
0503 13:58:23.121141 (+ 41624us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.295757 (+174616us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.366848 (+ 71091us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:23.300446 9519 rpcz_store.cc:276] 0503 13:58:21.014790 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.934206 (+1919416us) service_pool.cc:224] Handling call
0503 13:58:22.934510 (+ 304us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:22.934686 (+ 176us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:22.998155 (+ 63469us) log.cc:844] Serialized 14 byte log entry
0503 13:58:23.050547 (+ 52392us) spinlock_profiling.cc:243] Waited 6.39 ms on lock 0x7b1800219ca0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea8db62 00007f9f3ea8fe01 00007f9f3ea8fdb9 00007f9f3ea8fd49 00007f9f3ea8fd11 00007f9f3ea8f00d 0000000000379254 0000000000379099 00007f9f3ea70fba
0503 13:58:23.124231 (+ 73684us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:23.125998 (+ 1767us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:23.133074 (+ 7076us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.222161 (+ 89087us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:23.241200 (+ 19039us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":587,"prepare.queue_time_us":46471,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":21687,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":6123,"spinlock_wait_cycles":220862720,"thread_start_us":80089,"threads_started":1,"wal-append.queue_time_us":131738}
I20260503 13:58:23.551497 10291 raft_consensus.cc:515] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.721745 9566 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:23.656855 9449 raft_consensus.cc:1240] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:23.576454 10171 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:23.787992 10204 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.736s user 0.004s sys 0.007s
W20260503 13:58:23.810941 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 89299.4us
W20260503 13:58:23.811935 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 399823us
I20260503 13:58:23.738010 9527 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:23.728240 10302 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:23.729579 10304 raft_consensus.cc:3055] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Stepping down as leader of term 2
W20260503 13:58:23.734457 10089 consensus_meta.cc:287] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.892s user 0.004s sys 0.007s
W20260503 13:58:23.736136 10113 consensus_meta.cc:287] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.797s user 0.003s sys 0.009s
W20260503 13:58:23.816315 9565 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:23.734859 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:23.716625 9444 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.002s sys 0.007s
I20260503 13:58:23.599669 10296 raft_consensus.cc:515] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.732266 10179 raft_consensus.cc:3055] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:23.725162 10251 raft_consensus.cc:740] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
W20260503 13:58:23.876395 9444 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.753s user 0.005s sys 0.007s
I20260503 13:58:23.631081 10079 raft_consensus.cc:515] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.726022 10196 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:23.815585 9601 raft_consensus.cc:3055] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:23.884892 9601 raft_consensus.cc:740] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:23.784308 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:23.631214 10200 consensus_queue.cc:237] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:23.673986 9974 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s user 0.002s sys 0.007s
I20260503 13:58:23.817888 10304 raft_consensus.cc:740] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
W20260503 13:58:23.836072 10091 consensus_meta.cc:287] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.759s user 0.004s sys 0.000s
I20260503 13:58:23.838507 10240 leader_election.cc:290] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:23.829707 10171 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:23.815101 10283 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:23.819877 10295 consensus_queue.cc:1059] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
W20260503 13:58:23.674600 9643 ts_tablet_manager.cc:1469] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868: Trace:
0503 13:58:05.687731 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.224847 (+537116us) service_pool.cc:224] Handling call
0503 13:58:06.239355 (+ 14508us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.239439 (+ 84us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.265245 (+ 25806us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.272663 (+ 7418us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.273146 (+ 483us) inbound_call.cc:177] Queueing success response
0503 13:58:21.865027 (+15591881us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0503 13:58:22.879114 (+1014087us) ts_tablet_manager.cc:1435] Starting tablet replica
0503 13:58:23.002426 (+123312us) tablet_replica.cc:261] Starting instrumentation
0503 13:58:23.002897 (+ 471us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":39933,"raft.queue_time_us":182223,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":15119,"tablet-open.queue_time_us":15592003,"thread_start_us":457144,"threads_started":1}
I20260503 13:58:23.807569 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:23.830945 10291 leader_election.cc:290] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:23.725862 10202 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:23.829753 10191 consensus_meta.cc:287] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.555s user 0.007s sys 0.000s
I20260503 13:58:23.904734 10191 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.905670 9453 raft_consensus.cc:1240] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:23.905778 10079 leader_election.cc:290] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:23.906510 10191 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:23.906291 10304 consensus_queue.cc:260] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:23.907262 9600 consensus_meta.cc:287] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 1.017s user 0.013s sys 0.000s
I20260503 13:58:23.907564 10304 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
W20260503 13:58:23.908368 10017 consensus_meta.cc:287] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.843s user 0.001s sys 0.006s
I20260503 13:58:23.909009 10202 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:23.906512 9601 consensus_queue.cc:260] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.909587 9601 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:23.905730 10296 leader_election.cc:290] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:23.907675 9600 raft_consensus.cc:2468] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
W20260503 13:58:23.915421 10099 consensus_meta.cc:287] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.678s user 0.005s sys 0.000s
W20260503 13:58:23.840539 10243 consensus_meta.cc:287] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.859s user 0.002s sys 0.003s
I20260503 13:58:23.896081 10251 consensus_queue.cc:260] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.925133 10251 raft_consensus.cc:3060] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:23.927614 9642 tablet_bootstrap.cc:654] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:23.876518 10179 raft_consensus.cc:740] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:23.929839 10179 consensus_queue.cc:260] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.896512 9444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 296) took 2073 ms. Trace:
I20260503 13:58:23.896415 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:23.903601 9593 raft_consensus.cc:2393] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:23.900334 10287 raft_consensus.cc:2749] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:23.932090 9444 rpcz_store.cc:276] 0503 13:58:21.834185 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.027614 (+1193429us) service_pool.cc:224] Handling call
0503 13:58:23.028147 (+ 533us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.028325 (+ 178us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:23.087231 (+ 58906us) log.cc:844] Serialized 14 byte log entry
0503 13:58:23.889531 (+802300us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:23.896227 (+ 6696us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:23.896275 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.896341 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:23.896460 (+ 119us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":192648,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":4206,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":1223,"spinlock_wait_cycles":28602624,"thread_start_us":807921,"threads_started":2,"wal-append.queue_time_us":630891}
I20260503 13:58:23.815469 10188 consensus_queue.cc:260] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.930821 10302 raft_consensus.cc:2749] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:23.930872 10179 raft_consensus.cc:3060] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:23.820844 9898 consensus_queue.cc:237] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.916914 9598 raft_consensus.cc:1275] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:23.941499 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:23.942871 9598 raft_consensus.cc:1275] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260503 13:58:23.943209 9598 pending_rounds.cc:85] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868: Aborting all ops after (but not including) 0
I20260503 13:58:23.943428 9598 pending_rounds.cc:107] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868: Aborting uncommitted NO_OP operation due to leader change: 1.1
W20260503 13:58:23.842271 10238 consensus_meta.cc:287] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.717s user 0.006s sys 0.000s
I20260503 13:58:23.936463 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:23.944466 9593 raft_consensus.cc:2410] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:23.935865 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:23.939081 9450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 270) took 4105 ms. Trace:
I20260503 13:58:23.954903 9450 rpcz_store.cc:276] 0503 13:58:19.837263 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.921198 (+3083935us) service_pool.cc:224] Handling call
0503 13:58:22.922574 (+ 1376us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.688651 (+766077us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:23.938886 (+250235us) spinlock_profiling.cc:243] Waited 612 ms on lock 0x7b54002945d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:23.939031 (+ 145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1408354176}
I20260503 13:58:23.921830 9643 tablet_bootstrap.cc:492] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:23.964272 9597 raft_consensus.cc:1275] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:23.964794 10163 consensus_queue.cc:1059] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:58:23.965560 9597 raft_consensus.cc:1275] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:23.966015 10303 raft_consensus.cc:3055] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:23.966372 10303 raft_consensus.cc:740] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:23.966794 9597 raft_consensus.cc:1275] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260503 13:58:23.967622 10309 log.cc:927] Time spent T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.239s user 0.007s sys 0.001s
I20260503 13:58:23.967089 10303 consensus_queue.cc:260] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:23.965160 9418 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.891s user 0.008s sys 0.015s
W20260503 13:58:23.968266 9418 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.894s user 0.008s sys 0.015s
W20260503 13:58:23.968147 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 147461us
W20260503 13:58:23.968590 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 912958us
W20260503 13:58:23.969414 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 156647us
I20260503 13:58:23.943678 9598 raft_consensus.cc:2889] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:23.916200 9601 raft_consensus.cc:1275] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:23.970062 9492 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:23.976827 9567 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.005s sys 0.009s
W20260503 13:58:23.978535 9567 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.548s user 0.005s sys 0.009s
W20260503 13:58:23.978837 9418 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:23.979341 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 589531us
W20260503 13:58:23.979545 9418 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:23.980094 9418 leader_election.cc:343] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:23.980379 9418 leader_election.cc:304] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:23.975857 10113 consensus_queue.cc:1059] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260503 13:58:23.981763 10287 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:23.982201 10287 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.982601 10039 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:23.983002 10039 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:23.983929 9598 raft_consensus.cc:1240] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:23.984277 10287 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:23.984905 10238 raft_consensus.cc:2749] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:23.986377 10238 consensus_queue.cc:1048] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:23.987632 10287 consensus_queue.cc:1048] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:23.989398 10238 consensus_queue.cc:1048] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:23.989773 10091 raft_consensus.cc:3055] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:23.990137 10091 raft_consensus.cc:740] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:23.990810 10091 consensus_queue.cc:260] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:23.991041 10238 consensus_queue.cc:1048] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:23.991889 10091 raft_consensus.cc:3060] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:23.993707 10188 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:23.997094 10295 consensus_queue.cc:1059] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:23.997745 10296 consensus_queue.cc:1059] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:58:23.997869 10248 consensus_queue.cc:1059] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:23.998462 10296 consensus_queue.cc:1059] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20260503 13:58:24.024927 9491 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:23.986344 9598 raft_consensus.cc:1275] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:23.998509 10307 consensus_queue.cc:1048] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20260503 13:58:24.059571 10241 consensus_meta.cc:287] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.810s user 0.004s sys 0.004s
I20260503 13:58:23.998914 10295 raft_consensus.cc:3055] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:23.998632 10248 consensus_queue.cc:1243] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer 1cb63886040047559d3893923cc9aaf6 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20260503 13:58:24.064785 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 66517.6us
W20260503 13:58:24.066111 9445 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.013s sys 0.029s
W20260503 13:58:24.061045 10317 log.cc:927] Time spent T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
W20260503 13:58:24.067255 9445 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s user 0.014s sys 0.030s
I20260503 13:58:24.067989 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 303) took 1440 ms. Trace:
I20260503 13:58:24.062713 10288 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:24.068331 9445 rpcz_store.cc:276] 0503 13:58:22.627511 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.257838 (+630327us) service_pool.cc:224] Handling call
0503 13:58:23.276739 (+ 18901us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.276925 (+ 186us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:23.311627 (+ 34702us) log.cc:844] Serialized 14 byte log entry
0503 13:58:24.067667 (+756040us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:24.067712 (+ 45us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:24.067747 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:24.067816 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:24.067929 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42700,"prepare.queue_time_us":67115,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":38,"raft.queue_time_us":71,"thread_start_us":729221,"threads_started":1,"wal-append.queue_time_us":730338}
W20260503 13:58:24.112201 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 142201us
I20260503 13:58:24.113302 9492 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:24.067042 10295 raft_consensus.cc:740] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
W20260503 13:58:24.381032 9567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:24.410745 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:24.288415 9416 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:24.114235 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 46654.4us
I20260503 13:58:24.253875 9644 tablet_bootstrap.cc:492] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:24.313660 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 306) took 1685 ms. Trace:
I20260503 13:58:24.412976 9644 ts_tablet_manager.cc:1403] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 1.652s user 0.027s sys 0.014s
W20260503 13:58:24.310897 9493 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.012s sys 0.029s
W20260503 13:58:24.414047 9493 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.861s user 0.013s sys 0.034s
I20260503 13:58:24.242159 9643 tablet_bootstrap.cc:654] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:24.412909 9453 rpcz_store.cc:276] 0503 13:58:22.628182 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.267649 (+639467us) service_pool.cc:224] Handling call
0503 13:58:23.267887 (+ 238us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.906034 (+638147us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:24.313414 (+407380us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b54001d33d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:24.313602 (+ 188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1466384768}
I20260503 13:58:24.067395 10241 raft_consensus.cc:2749] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260503 13:58:24.421247 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 132895us
W20260503 13:58:24.425752 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:24.070575 10283 raft_consensus.cc:2749] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:24.169868 10302 raft_consensus.cc:2804] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:24.443764 10307 consensus_queue.cc:1059] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:24.444630 10188 consensus_queue.cc:1048] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:24.445183 10289 raft_consensus.cc:3055] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:24.445534 10289 raft_consensus.cc:740] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:24.446084 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:24.446249 10289 consensus_queue.cc:260] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.447311 10289 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:24.447916 9447 raft_consensus.cc:1275] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:24.436977 9416 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:24.189507 9494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:24.294375 10028 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.111948 10287 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.365034 10310 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.411780 10243 raft_consensus.cc:3055] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:24.440150 9601 raft_consensus.cc:1240] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:24.441435 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:24.276968 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:24.260255 10238 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.294701 9978 consensus_queue.cc:260] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.420681 10091 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.428833 9449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 260) took 2623 ms. Trace:
I20260503 13:58:24.428803 10179 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.442296 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:24.420838 10170 raft_consensus.cc:3060] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:24.225643 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:24.443166 9444 raft_consensus.cc:1240] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
W20260503 13:58:24.253012 10321 log.cc:927] Time spent T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260503 13:58:24.284958 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:24.433694 9491 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:24.475116 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:24.410545 10253 raft_consensus.cc:3055] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:24.261559 10318 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:24.449687 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 336440us
I20260503 13:58:24.312800 10166 raft_consensus.cc:3055] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:24.315148 10300 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.459578 10319 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.459646 10251 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
W20260503 13:58:24.480082 9491 leader_election.cc:343] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:24.451678 9416 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:24.453949 9566 leader_election.cc:343] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:24.424283 10200 raft_consensus.cc:2749] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260503 13:58:24.417306 9527 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 247) took 1660 ms (client timeout 2042 ms). Trace:
I20260503 13:58:24.314950 9600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 259) took 1896 ms. Trace:
W20260503 13:58:24.458972 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:24.429037 10296 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.429200 10304 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.461773 9450 raft_consensus.cc:2393] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:24.275077 10039 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.445070 10113 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.462271 9644 raft_consensus.cc:359] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.464955 10091 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.463941 9449 rpcz_store.cc:276] 0503 13:58:21.800836 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.026218 (+1225382us) service_pool.cc:224] Handling call
0503 13:58:23.026479 (+ 261us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.805386 (+778907us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:24.330491 (+525105us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b54001d2ed4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:24.424213 (+ 93722us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1318536576}
I20260503 13:58:24.443066 10295 consensus_queue.cc:260] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.452354 10199 raft_consensus.cc:2749] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:24.494853 9978 raft_consensus.cc:3060] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:24.458040 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:24.463292 10310 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.461267 10243 raft_consensus.cc:740] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:24.466500 9597 raft_consensus.cc:1240] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:24.489305 9492 leader_election.cc:304] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:24.490504 9416 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:24.494129 9644 raft_consensus.cc:385] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:24.516605 9644 raft_consensus.cc:740] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:24.457257 9898 raft_consensus.cc:2804] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:24.471457 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 270) took 1125 ms. Trace:
I20260503 13:58:24.460758 10287 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.459807 10028 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.504312 9526 raft_consensus.cc:1240] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:24.528645 10295 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:24.452798 10302 raft_consensus.cc:697] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:24.529090 10243 consensus_queue.cc:260] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.475879 10253 raft_consensus.cc:740] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
W20260503 13:58:24.494200 10308 log.cc:927] Time spent T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.764s user 0.008s sys 0.007s
I20260503 13:58:24.538091 10287 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:24.538602 10287 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:24.539011 10287 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.511482 9601 raft_consensus.cc:1275] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:58:24.540936 10039 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:24.512801 10313 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.541517 10260 raft_consensus.cc:3055] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:24.541864 10260 raft_consensus.cc:740] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:24.541672 10313 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.542572 10260 consensus_queue.cc:260] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:24.464650 9494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:24.489382 10166 raft_consensus.cc:740] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:24.549275 10166 consensus_queue.cc:260] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.489854 10184 raft_consensus.cc:493] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.467063 10238 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.495777 10304 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:24.519630 9416 leader_election.cc:343] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:24.523236 10307 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.497756 10314 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.520226 9644 consensus_queue.cc:260] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.516755 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 297) took 2665 ms. Trace:
I20260503 13:58:24.524341 10200 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.495081 10113 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.524701 9594 rpcz_store.cc:276] 0503 13:58:23.345626 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.380246 (+ 34620us) service_pool.cc:224] Handling call
0503 13:58:23.380600 (+ 354us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:23.767715 (+387115us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:23.784487 (+ 16772us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:23.836912 (+ 52425us) log.cc:844] Serialized 26 byte log entry
0503 13:58:24.002049 (+165137us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:24.005639 (+ 3590us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:24.062118 (+ 56479us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:24.461951 (+399833us) raft_consensus.cc:1685] finished
0503 13:58:24.463008 (+ 1057us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:24.469529 (+ 6521us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39196,"raft.queue_time_us":227583,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":91,"thread_start_us":433642,"threads_started":2,"wal-append.queue_time_us":225429}
I20260503 13:58:24.500546 9991 raft_consensus.cc:2749] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:24.490938 10188 consensus_queue.cc:1048] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260503 13:58:24.500386 10163 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:24.500758 10324 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:24.529850 10302 consensus_queue.cc:237] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.535977 9598 raft_consensus.cc:1275] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260503 13:58:24.536634 10303 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:24.489727 9600 rpcz_store.cc:276] 0503 13:58:22.418924 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.741164 (+322240us) service_pool.cc:224] Handling call
0503 13:58:24.314778 (+1573614us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b5400059454. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:24.314889 (+ 111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":333609728}
W20260503 13:58:24.494700 9527 rpcz_store.cc:269] 0503 13:58:22.756478 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:22.774398 (+ 17920us) service_pool.cc:224] Handling call
0503 13:58:24.325105 (+1550707us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b540004e554. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:24.411765 (+ 86660us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2066844032}
I20260503 13:58:24.492782 10251 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.537336 10028 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:24.544404 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:24.536007 10091 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:24.546334 10287 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:24.515434 10089 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.495110 10204 consensus_queue.cc:1048] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
I20260503 13:58:24.550580 9566 leader_election.cc:304] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:24.633160 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 82667.2us
I20260503 13:58:24.542361 10193 consensus_queue.cc:1059] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:58:24.554270 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:24.517063 9599 raft_consensus.cc:3060] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:24.471154 10179 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.490566 10318 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:24.514497 10283 consensus_queue.cc:1175] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "428b342dc4f041df956015e7d7cd173a" responder_term: 2 status { last_received { term: 2 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 2 index: 1 } }
W20260503 13:58:24.579450 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 59852.3us
I20260503 13:58:24.520795 9528 raft_consensus.cc:1275] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:24.579830 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:24.521831 10199 raft_consensus.cc:2749] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:24.523618 10288 raft_consensus.cc:2749] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:24.532485 10243 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:24.497119 10296 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.543381 10260 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:24.493310 10300 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:24.618216 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:24.609053 9453 raft_consensus.cc:1275] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:24.580262 10241 raft_consensus.cc:2804] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:24.494132 10319 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.609141 10200 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:24.631259 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 142018us
I20260503 13:58:24.635958 9418 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:24.687033 9492 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
W20260503 13:58:24.687067 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 51178us
I20260503 13:58:24.605643 10159 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.640348 9642 tablet_bootstrap.cc:492] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:24.627431 9519 raft_consensus.cc:1240] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:24.628118 10099 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.582574 10184 raft_consensus.cc:515] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.662001 9526 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:24.661912 9416 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:24.662178 9644 ts_tablet_manager.cc:1434] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.249s user 0.008s sys 0.000s
I20260503 13:58:24.605445 10039 consensus_queue.cc:1048] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
W20260503 13:58:24.639019 10306 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.910s user 0.027s sys 0.051s
I20260503 13:58:24.661232 9444 raft_consensus.cc:1240] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:24.631876 9898 raft_consensus.cc:697] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:24.636225 10028 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.751267 10028 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.752203 9593 raft_consensus.cc:1275] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:24.637529 10089 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.753728 10028 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:24.754089 10089 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:24.754573 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:24.754963 10251 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:24.663048 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:24.755247 10297 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.755926 10238 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:24.755749 10297 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.756124 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:24.603407 10307 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.629360 9601 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:24.630091 10163 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.665442 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:24.714345 9642 ts_tablet_manager.cc:1403] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 1.587s user 0.028s sys 0.038s
W20260503 13:58:24.715833 10326 log.cc:927] Time spent T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.206s user 0.000s sys 0.001s
I20260503 13:58:24.606045 10314 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.717340 10318 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:24.718169 10179 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:24.718336 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:24.718725 9644 tablet_bootstrap.cc:492] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:24.718641 10337 consensus_queue.cc:1048] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:24.720165 10184 leader_election.cc:290] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:24.720352 10300 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:24.720538 10099 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:24.720652 10330 log.cc:927] Time spent T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20260503 13:58:24.641783 10315 raft_consensus.cc:2749] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260503 13:58:24.714174 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 52338.8us
I20260503 13:58:24.721886 10193 consensus_queue.cc:1048] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:24.722368 10310 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:24.724792 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:24.688505 10340 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.630447 10324 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.750340 9898 consensus_queue.cc:237] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.753656 10113 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:24.723161 9643 tablet_bootstrap.cc:492] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:24.899120 9643 ts_tablet_manager.cc:1403] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.978s user 0.025s sys 0.055s
I20260503 13:58:24.688580 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:24.760877 9491 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:24.727530 9991 raft_consensus.cc:2749] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:24.755201 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:24.888010 9525 raft_consensus.cc:2393] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:24.783898 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:24.904724 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 143916us
I20260503 13:58:24.905313 9451 raft_consensus.cc:2468] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
W20260503 13:58:24.890244 9416 leader_election.cc:343] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:24.787297 10200 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:24.891835 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:24.893402 9450 raft_consensus.cc:2468] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:24.663098 9449 raft_consensus.cc:1240] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:24.713624 10287 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:24.723404 10306 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.994s user 0.027s sys 0.052s
I20260503 13:58:24.712867 10199 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:24.721693 10296 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:24.721745 10304 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:24.724385 10241 raft_consensus.cc:697] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:24.723495 10313 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:24.782301 9526 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:24.904143 9528 raft_consensus.cc:2410] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:24.784699 10204 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.787509 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:24.785236 10238 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
W20260503 13:58:24.907266 9416 leader_election.cc:343] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:24.890946 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 132668us
I20260503 13:58:24.785463 10332 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.788834 10342 raft_consensus.cc:493] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.899173 10283 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.889734 9452 raft_consensus.cc:2468] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:24.899881 10315 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.898828 10339 raft_consensus.cc:2804] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:24.929575 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:24.898618 10251 consensus_queue.cc:1059] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260503 13:58:24.901173 9594 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:24.934629 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:24.935381 10099 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:24.937109 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:24.929289 10335 raft_consensus.cc:2804] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:24.930263 10287 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.928135 10199 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:24.951591 9447 raft_consensus.cc:2468] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:24.939390 10324 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:24.897033 10292 consensus_queue.cc:1059] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260503 13:58:24.945390 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:24.888469 10159 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.908607 10188 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.948146 9566 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:24.948343 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:24.949648 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:24.972435 10283 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.910573 9519 raft_consensus.cc:2468] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
W20260503 13:58:24.967085 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 241370us
I20260503 13:58:24.921519 10318 consensus_queue.cc:1048] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260503 13:58:24.976269 9451 raft_consensus.cc:2468] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:24.977664 9418 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
W20260503 13:58:25.021502 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 43914us
I20260503 13:58:24.966476 10342 raft_consensus.cc:515] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.905434 10340 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.985204 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:24.938540 10337 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.957214 10307 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:24.965451 9596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 240) took 1414 ms. Trace:
I20260503 13:58:24.965590 10319 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:24.951437 10238 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.964644 10310 consensus_queue.cc:1048] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260503 13:58:25.028750 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:24.965164 9991 consensus_queue.cc:1048] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260503 13:58:24.959369 10332 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:24.946527 10204 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:24.992388 10170 consensus_meta.cc:287] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.517s user 0.008s sys 0.000s
I20260503 13:58:24.970896 10184 consensus_queue.cc:1048] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260503 13:58:24.970996 10300 consensus_queue.cc:1048] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
W20260503 13:58:25.013731 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 65651.6us
I20260503 13:58:24.987928 10339 raft_consensus.cc:697] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:24.923421 10113 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.985195 9599 raft_consensus.cc:2468] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:24.991572 10199 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:25.019716 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:24.980304 9642 raft_consensus.cc:359] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.039000 9566 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:25.019569 10188 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.028303 10342 leader_election.cc:290] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.030082 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.030190 9418 leader_election.cc:304] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:24.990876 10352 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:24.602473 9446 rpcz_store.cc:276] 0503 13:58:21.849117 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.030027 (+1180910us) service_pool.cc:224] Handling call
0503 13:58:23.074316 (+ 44289us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:23.086255 (+ 11939us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:24.278476 (+1192221us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b18000b3c20. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f416d285c 00007f9f3ea70c0b 00007f9f47869bc9 00007f9f3ea8ccdb 00007f9f3ea8af7f 00007f9f3ea8aa95 00007f9f4552e837 00007f9f41b35d87 00007f9f41b4549f
0503 13:58:24.278632 (+ 156us) log.cc:844] Serialized 14 byte log entry
0503 13:58:24.470246 (+191614us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:24.475826 (+ 5580us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:24.479948 (+ 4122us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:24.497420 (+ 17472us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:24.514116 (+ 16696us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":175669,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":3903,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":6487,"spinlock_wait_cycles":9072384,"thread_start_us":318207,"threads_started":2,"wal-append.queue_time_us":303971}
I20260503 13:58:24.986366 10335 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.020131 10347 raft_consensus.cc:2804] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:25.017262 10163 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.080358 10163 consensus_queue.cc:1059] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:25.082546 9643 raft_consensus.cc:359] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.083173 9643 raft_consensus.cc:385] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:25.083410 9643 raft_consensus.cc:740] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:25.083961 9643 consensus_queue.cc:260] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:25.084702 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 45778.7us
I20260503 13:58:25.058158 10113 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.085647 10340 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.085806 9566 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:25.087050 9566 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:25.088146 10304 raft_consensus.cc:2804] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:25.021502 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:24.989161 10315 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.039374 9642 raft_consensus.cc:385] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:25.040047 10170 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:25.013666 10200 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:25.030992 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:25.088966 10295 consensus_meta.cc:287] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.548s user 0.008s sys 0.001s
I20260503 13:58:25.050323 9416 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:25.124771 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 74529.1us
I20260503 13:58:25.056931 9445 raft_consensus.cc:2468] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:24.994804 10241 consensus_queue.cc:237] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.061498 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:25.037245 10289 consensus_meta.cc:287] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.590s user 0.009s sys 0.000s
I20260503 13:58:25.088752 9525 raft_consensus.cc:2468] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:25.057797 10017 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.094810 9642 raft_consensus.cc:740] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:25.136893 9642 consensus_queue.cc:260] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.095716 10342 raft_consensus.cc:2804] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:25.089681 10347 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.040757 10337 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
W20260503 13:58:25.121160 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:25.032450 10028 raft_consensus.cc:3055] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:25.094273 10300 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.123436 10358 raft_consensus.cc:3055] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:25.040723 9519 raft_consensus.cc:2468] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:25.137639 10318 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:25.041054 10346 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.068979 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 242) took 1450 ms. Trace:
I20260503 13:58:25.127089 10170 raft_consensus.cc:697] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:25.133257 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.133666 10283 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.120677 10302 raft_consensus.cc:493] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.107559 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.116721 10319 raft_consensus.cc:2804] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:25.097201 10340 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.035967 10349 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.142074 10315 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.093770 10335 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.096002 10304 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.126636 10202 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.076884 10352 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.144126 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.156487 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.137372 10324 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.154407 9643 ts_tablet_manager.cc:1434] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.255s user 0.005s sys 0.000s
I20260503 13:58:25.166275 9416 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:25.233511 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 67322.9us
I20260503 13:58:25.132524 10314 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.125439 10200 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.139249 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.139454 10342 raft_consensus.cc:493] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.163332 10295 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.030732 9596 rpcz_store.cc:276] 0503 13:58:23.551174 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.902817 (+351643us) service_pool.cc:224] Handling call
0503 13:58:23.903138 (+ 321us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:23.988298 (+ 85160us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:23.988329 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:23.988925 (+ 596us) log.cc:844] Serialized 26 byte log entry
0503 13:58:24.121340 (+132415us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:24.121399 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:24.121491 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:24.941955 (+820464us) raft_consensus.cc:1685] finished
0503 13:58:24.945902 (+ 3947us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:24.957863 (+ 11961us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":84802,"raft.queue_time_us":84988,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":81740,"threads_started":1,"wal-append.queue_time_us":475415}
I20260503 13:58:25.201107 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.127736 9601 raft_consensus.cc:2468] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:25.213670 9595 raft_consensus.cc:2410] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260503 13:58:25.163801 10287 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.253062 9991 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.146160 10347 raft_consensus.cc:3060] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.232352 9642 ts_tablet_manager.cc:1434] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.450s user 0.007s sys 0.000s
W20260503 13:58:25.172482 9494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:25.252295 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 221281us
I20260503 13:58:25.164832 10358 raft_consensus.cc:740] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:25.253504 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:25.214427 10303 consensus_meta.cc:287] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.584s user 0.001s sys 0.007s
I20260503 13:58:25.245769 10355 raft_consensus.cc:493] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:25.252671 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 222532us
I20260503 13:58:25.140022 10017 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.254379 10342 raft_consensus.cc:3060] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:25.157490 10028 raft_consensus.cc:740] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:25.259811 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.243654 10302 raft_consensus.cc:515] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.199189 10334 raft_consensus.cc:2804] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:25.208333 9524 raft_consensus.cc:2468] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:25.214092 10159 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.260222 9991 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.207660 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.231501 10319 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.279935 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.231638 10304 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.158803 10300 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.238566 10340 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.236899 10184 raft_consensus.cc:2749] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:25.283411 9418 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:25.244485 10324 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.253929 10188 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.261166 10170 consensus_queue.cc:237] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.290942 9643 tablet_bootstrap.cc:492] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:25.243997 9527 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:25.172104 9599 rpcz_store.cc:276] 0503 13:58:23.618580 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:24.411900 (+793320us) service_pool.cc:224] Handling call
0503 13:58:25.058943 (+647043us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:25.308499 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.309002 9599 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
W20260503 13:58:25.200278 10260 consensus_meta.cc:287] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.511s user 0.005s sys 0.001s
I20260503 13:58:25.257727 9447 raft_consensus.cc:2468] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:25.311919 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.294390 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.172730 10346 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.289326 9519 raft_consensus.cc:2468] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:25.196790 10339 consensus_queue.cc:237] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.298455 10200 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.299072 10300 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.299865 9991 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.296453 10319 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:25.300508 9600 raft_consensus.cc:1240] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:25.300819 10334 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:25.301252 9597 raft_consensus.cc:2468] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:25.301687 10302 leader_election.cc:290] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.169425 10289 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.282220 9416 leader_election.cc:304] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:25.287034 10287 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.293483 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:25.302448 10243 consensus_meta.cc:287] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.636s user 0.006s sys 0.002s
I20260503 13:58:25.292150 10204 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.237042 10202 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.307291 9418 leader_election.cc:304] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:25.335671 10324 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.336838 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.337131 9418 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:25.237977 10349 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.339573 9491 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:25.303768 10159 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.296878 10199 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.295778 9526 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:25.296495 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.294337 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.396575 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.303283 9450 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:25.303265 10238 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.290517 10303 consensus_queue.cc:1048] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260503 13:58:25.427357 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:25.286705 10295 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.318550 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.429045 9599 raft_consensus.cc:2468] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:25.319562 9566 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:25.312295 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.310444 10184 raft_consensus.cc:2804] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:25.290069 10314 consensus_queue.cc:1048] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
W20260503 13:58:25.330691 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 68485.5us
I20260503 13:58:25.331190 10289 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.290210 10306 consensus_queue.cc:1059] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.024s
I20260503 13:58:25.313704 10332 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.377085 9418 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:25.322960 10334 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
W20260503 13:58:25.437114 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 117617us
W20260503 13:58:25.386605 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:25.358692 10260 raft_consensus.cc:2804] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260503 13:58:25.393498 10359 log.cc:927] Time spent T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20260503 13:58:25.424058 9642 tablet_bootstrap.cc:492] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
W20260503 13:58:25.427346 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 87840.8us
I20260503 13:58:25.434001 9416 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:25.434621 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 57621us
I20260503 13:58:25.438977 9449 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:25.394233 10300 raft_consensus.cc:2804] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:25.368922 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.442009 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.375986 9991 raft_consensus.cc:2749] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:25.444024 9418 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:25.444861 10238 raft_consensus.cc:2804] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:25.448838 10238 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.449167 10238 raft_consensus.cc:3060] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.452646 10243 consensus_queue.cc:1048] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:58:25.453914 10300 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.446954 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.316465 10028 consensus_queue.cc:260] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:25.459013 9567 leader_election.cc:343] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:25.467351 9416 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:25.467613 9567 leader_election.cc:304] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:25.459013 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.454315 10300 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:25.460325 10253 consensus_queue.cc:260] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.460836 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.461221 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.440641 10204 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.436436 10184 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.462167 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.462371 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.459801 10289 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.364399 10287 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.463207 9566 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:25.484484 9492 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:25.465718 10332 raft_consensus.cc:2749] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:25.466463 9644 tablet_bootstrap.cc:654] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:25.454474 10224 raft_consensus.cc:2804] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:25.470644 9567 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:25.441010 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.480711 10184 raft_consensus.cc:3060] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.441109 10260 raft_consensus.cc:493] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.476197 9526 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:25.431231 9596 raft_consensus.cc:2468] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:25.477787 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.398703 9594 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:25.362771 10349 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.479779 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.448249 10361 raft_consensus.cc:2804] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:25.479143 9416 leader_election.cc:304] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:25.394807 10159 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.482805 10357 raft_consensus.cc:2749] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:25.469388 9527 raft_consensus.cc:2468] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:25.473948 10352 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.491837 10091 raft_consensus.cc:2804] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:25.483486 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.491603 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.491914 9567 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:25.383610 9524 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:25.476627 9519 raft_consensus.cc:2468] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:25.294682 10355 raft_consensus.cc:515] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.482116 10310 raft_consensus.cc:2804] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:25.321321 10358 consensus_queue.cc:260] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.497402 10340 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.497424 9528 raft_consensus.cc:2468] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:25.492686 10289 raft_consensus.cc:2804] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:25.490324 10224 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.504403 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.503325 10361 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.504914 9418 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:25.480854 10204 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.507751 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.495545 10260 raft_consensus.cc:3060] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
W20260503 13:58:25.500854 9416 consensus_peers.cc:597] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 -> Peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Couldn't send request to peer 4bea375a55034750a1f8804b0db97868. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260503 13:58:25.517217 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:25.517766 9416 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:25.503152 10332 raft_consensus.cc:2804] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:25.523240 10332 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.523602 10332 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.524083 9567 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:25.497649 10307 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:25.510022 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.512940 10346 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.525219 9519 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:25.525066 10307 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.527889 9599 raft_consensus.cc:2410] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260503 13:58:25.538206 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:25.548497 9978 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:25.548923 9978 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.549239 9978 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:25.555832 10357 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.556319 10357 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.558408 10357 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.558848 10357 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.559217 10357 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.563846 10357 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.531337 10260 raft_consensus.cc:515] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.503633 9448 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:25.572075 10361 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:25.504477 10091 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.503188 10362 raft_consensus.cc:2749] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:25.574146 9492 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:25.576393 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.575696 10366 raft_consensus.cc:2749] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:25.572609 10224 raft_consensus.cc:3060] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:25.573341 10340 raft_consensus.cc:2749] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260503 13:58:25.579430 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 61749.8us
I20260503 13:58:25.513753 10289 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.508499 9444 raft_consensus.cc:2468] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
W20260503 13:58:25.579210 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 55240.1us
W20260503 13:58:25.579746 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 74915.5us
I20260503 13:58:25.514003 10310 raft_consensus.cc:697] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:25.517093 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.512431 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:25.581951 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:25.581669 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.581625 9526 raft_consensus.cc:1275] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:25.580466 9416 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:25.573501 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.573839 10306 raft_consensus.cc:2749] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:25.649108 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.652639 9599 raft_consensus.cc:2468] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:25.515463 10363 raft_consensus.cc:2804] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:25.583963 9418 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:25.650774 9528 raft_consensus.cc:1240] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260503 13:58:25.650779 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 70380.4us
W20260503 13:58:25.655048 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 71127.6us
I20260503 13:58:25.655695 9416 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:25.650261 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 256) took 1146 ms. Trace:
I20260503 13:58:25.574772 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.653363 9492 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:25.582973 10346 raft_consensus.cc:493] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.578682 10159 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.656157 9526 rpcz_store.cc:276] 0503 13:58:24.504027 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.516526 (+1012499us) service_pool.cc:224] Handling call
0503 13:58:25.581348 (+ 64822us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:25.650025 (+ 68677us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:25.650213 (+ 188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:25.657598 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 259) took 1152 ms. Trace:
I20260503 13:58:25.660940 9528 rpcz_store.cc:276] 0503 13:58:24.504712 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.577160 (+1072448us) service_pool.cc:224] Handling call
0503 13:58:25.650563 (+ 73403us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:25.657347 (+ 6784us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:25.657552 (+ 205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:25.575263 10091 raft_consensus.cc:3060] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.579321 10339 raft_consensus.cc:2749] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:25.581779 10199 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.583789 10289 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.649243 10307 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.580348 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.657397 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.652805 10362 raft_consensus.cc:2804] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:25.651975 10260 leader_election.cc:290] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.666076 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.657960 10363 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260503 13:58:25.659356 9416 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.26.195:39997 timed out after 2.185s (SENT) [suppressed 7 similar messages]
I20260503 13:58:25.668320 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:25.669512 10370 log.cc:927] Time spent T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260503 13:58:25.658648 10340 raft_consensus.cc:2804] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:25.660612 10349 raft_consensus.cc:2804] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:25.658212 9594 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:25.654748 9520 raft_consensus.cc:2468] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:25.662551 10295 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.663655 10372 raft_consensus.cc:2804] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:25.668885 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.674515 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:25.676085 9418 leader_election.cc:343] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:25.660961 9452 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:25.665555 10362 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.666903 10357 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.670135 10363 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:25.684989 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.685817 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.686322 10184 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.686594 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.671553 10340 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:25.662660 10346 raft_consensus.cc:515] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.690074 9492 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:25.661240 10319 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.692687 9521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 329) took 1190 ms. Trace:
I20260503 13:58:25.706662 10346 leader_election.cc:290] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.705653 9521 rpcz_store.cc:276] 0503 13:58:24.502583 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:24.507901 (+ 5318us) service_pool.cc:224] Handling call
0503 13:58:25.306237 (+798336us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b5800004598. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50a3e 000000000035a3c9 00007f9f46a888e0 00007f9f46b51784 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:25.306411 (+ 174us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:25.323407 (+ 16996us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:25.329914 (+ 6507us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:25.473598 (+143684us) log.cc:844] Serialized 52 byte log entry
0503 13:58:25.523152 (+ 49554us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:25.523207 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:25.523293 (+ 86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:25.692390 (+169097us) raft_consensus.cc:1685] finished
0503 13:58:25.692413 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:25.692632 (+ 219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64348,"raft.queue_time_us":5651,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":20164,"spinlock_wait_cycles":8396544,"thread_start_us":22574,"threads_started":1,"wal-append.queue_time_us":168808}
I20260503 13:58:25.707203 9593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 243) took 1970 ms. Trace:
I20260503 13:58:25.707589 9593 rpcz_store.cc:276] 0503 13:58:23.737271 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.962784 (+225513us) service_pool.cc:224] Handling call
0503 13:58:23.963069 (+ 285us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:24.752652 (+789583us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:25.706977 (+954325us) spinlock_profiling.cc:243] Waited 789 ms on lock 0x7b54000613d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:25.707145 (+ 168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1813916544}
I20260503 13:58:25.672976 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.708645 10319 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.708902 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:25.709004 9444 raft_consensus.cc:2468] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:25.709667 9593 raft_consensus.cc:2393] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:25.700214 10184 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.671969 10349 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.696161 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:25.687144 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:25.687307 10238 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.697415 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.716109 9526 raft_consensus.cc:2410] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:25.698067 9642 tablet_bootstrap.cc:654] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:25.701403 10297 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.707145 10303 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:25.691267 10372 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.690557 10355 leader_election.cc:290] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.711109 9492 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:25.717921 9596 raft_consensus.cc:3060] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:25.707604 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:25.721750 9492 leader_election.cc:343] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:25.694255 10304 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:25.711750 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 52409.7us
I20260503 13:58:25.712853 9594 raft_consensus.cc:2468] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:25.695689 10357 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.709935 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.679589 10287 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.690896 10362 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:25.720901 10372 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.703868 10340 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:25.665118 10366 consensus_queue.cc:1048] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260503 13:58:25.724687 9528 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:25.716087 10349 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:25.726874 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:25.726668 9521 raft_consensus.cc:3060] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.718133 10303 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.694146 10347 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.711913 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.738909 9602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 280) took 1257 ms. Trace:
I20260503 13:58:25.739265 9602 rpcz_store.cc:276] 0503 13:58:24.481039 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:24.489665 (+ 8626us) service_pool.cc:224] Handling call
0503 13:58:24.516014 (+ 26349us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:24.664506 (+148492us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:24.681686 (+ 17180us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:24.748969 (+ 67283us) log.cc:844] Serialized 26 byte log entry
0503 13:58:24.891972 (+143003us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:24.892018 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:25.738561 (+846543us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b5400038754. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:25.738613 (+ 52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:25.738659 (+ 46us) raft_consensus.cc:1685] finished
0503 13:58:25.738675 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:25.738855 (+ 180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52217,"raft.queue_time_us":28080,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":17832,"spinlock_wait_cycles":248640256,"thread_start_us":142657,"threads_started":1,"wal-append.queue_time_us":142973}
I20260503 13:58:25.740768 10238 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.741600 9447 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:25.742467 10253 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:25.743372 9596 raft_consensus.cc:2468] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:25.729521 9416 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:25.671947 9527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 262) took 1166 ms. Trace:
I20260503 13:58:25.725888 10297 raft_consensus.cc:2804] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:25.698339 10310 consensus_queue.cc:237] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.726056 10204 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.717350 10346 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.741070 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:25.733232 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:25.733423 9418 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:25.750365 9601 raft_consensus.cc:2393] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:25.748454 10355 raft_consensus.cc:493] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.736441 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:25.735458 10318 consensus_meta.cc:287] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.564s user 0.009s sys 0.000s
I20260503 13:58:25.756721 10260 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:25.736578 10303 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:25.749701 10287 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:25.746868 10297 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.726365 10319 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:25.751601 9528 raft_consensus.cc:3060] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.745393 9451 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:25.746811 9521 raft_consensus.cc:2468] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:25.771714 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.771840 9492 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:25.772248 9521 raft_consensus.cc:2468] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:25.773042 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.774001 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.775597 9528 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:25.775871 9567 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:25.776326 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:25.777220 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:25.732856 9978 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.747790 10337 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:25.762853 10304 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.756994 10113 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.763651 9522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 326) took 1261 ms. Trace:
I20260503 13:58:25.762317 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:25.759388 10335 consensus_meta.cc:287] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.505s user 0.005s sys 0.004s
I20260503 13:58:25.777901 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.759867 9594 raft_consensus.cc:2468] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:25.781325 10204 raft_consensus.cc:2804] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:25.768923 10260 raft_consensus.cc:697] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:25.746685 10300 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.778839 9418 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:25.784945 9521 raft_consensus.cc:2468] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:25.785246 9418 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:25.764645 10287 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.749504 10184 consensus_queue.cc:1048] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260503 13:58:25.759238 10318 raft_consensus.cc:2749] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:25.747473 9527 rpcz_store.cc:276] 0503 13:58:24.505359 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.513479 (+1008120us) service_pool.cc:224] Handling call
0503 13:58:25.670405 (+156926us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:25.763880 9643 tablet_bootstrap.cc:654] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:25.782721 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:25.751983 10355 raft_consensus.cc:515] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.769264 10297 raft_consensus.cc:3060] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.779021 10039 raft_consensus.cc:2804] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:25.757253 10346 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.757550 10224 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.785845 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.787725 9567 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:25.784529 10204 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.778721 10319 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.787930 9451 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.787014 10287 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
W20260503 13:58:25.792836 9418 leader_election.cc:343] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:25.795847 9418 leader_election.cc:304] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:25.788147 10366 raft_consensus.cc:2804] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:25.787079 10238 raft_consensus.cc:2804] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:25.784797 9596 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:25.798209 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.795537 10355 leader_election.cc:290] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.791364 10039 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:25.792383 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.820927 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.786671 10335 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.827047 9521 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:25.796952 10113 raft_consensus.cc:2804] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:25.827715 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.827909 9567 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:25.795827 10204 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.801730 10318 raft_consensus.cc:2749] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:25.832221 10315 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:25.832620 10315 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.832918 10315 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
W20260503 13:58:25.838383 9418 leader_election.cc:343] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:25.838362 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.839311 9521 raft_consensus.cc:2410] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:25.840054 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.840606 9521 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:25.818926 10357 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.841972 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:25.842964 9521 raft_consensus.cc:2393] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:25.843611 9567 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:25.843010 9526 raft_consensus.cc:1240] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260503 13:58:25.823275 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.830031 10113 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.828397 10039 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:25.826668 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.822072 10366 raft_consensus.cc:697] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:25.853134 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:25.854440 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.856415 9599 raft_consensus.cc:2393] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:25.841116 10347 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.855835 9527 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:25.859213 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.860184 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.846019 10304 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:25.847397 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.853614 10113 raft_consensus.cc:3060] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:25.864480 10339 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:25.782222 9522 rpcz_store.cc:276] 0503 13:58:24.501773 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:24.512714 (+ 10941us) service_pool.cc:224] Handling call
0503 13:58:24.537003 (+ 24289us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:24.908880 (+371877us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:24.910704 (+ 1824us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:24.924009 (+ 13305us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:24.929433 (+ 5424us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:25.763425 (+833992us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b540038d954. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:25.763478 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:25.763599 (+ 121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":832063744}
I20260503 13:58:25.822849 10260 consensus_queue.cc:237] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.871868 9567 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:25.870018 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.872581 9599 raft_consensus.cc:2468] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
W20260503 13:58:25.870105 10342 consensus_meta.cc:287] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.574s user 0.010s sys 0.000s
I20260503 13:58:25.873970 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.861963 10304 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:25.880690 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.873054 9596 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:25.887413 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.888046 10347 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:25.871883 9978 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.911461 9644 tablet_bootstrap.cc:492] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:25.911971 9644 ts_tablet_manager.cc:1403] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 1.193s user 0.034s sys 0.019s
I20260503 13:58:25.868255 10297 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.914012 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.914875 9599 raft_consensus.cc:2410] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:25.879932 10304 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:25.921007 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.919052 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 336) took 1123 ms. Trace:
I20260503 13:58:25.921865 9525 rpcz_store.cc:276] 0503 13:58:24.795331 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:24.930375 (+135044us) service_pool.cc:224] Handling call
0503 13:58:25.918880 (+988505us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b540004e054. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:25.918994 (+ 114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":251514112}
I20260503 13:58:25.879029 10346 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.881913 10300 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.888990 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:25.859488 10366 consensus_queue.cc:237] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.874486 10339 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.878927 10342 raft_consensus.cc:515] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.893113 10319 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:25.924741 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:25.913173 10335 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:25.890729 10334 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.004850 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.105926 9449 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:26.106168 9418 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:25.944164 9451 raft_consensus.cc:2468] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:25.943230 9566 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:25.950233 9453 raft_consensus.cc:2393] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
W20260503 13:58:26.108757 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 165601us
I20260503 13:58:26.111078 9566 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:26.112351 9566 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:25.967790 9445 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 310) took 1535 ms (client timeout 1950 ms). Trace:
W20260503 13:58:26.113104 9445 rpcz_store.cc:269] 0503 13:58:24.432180 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:24.442048 (+ 9868us) service_pool.cc:224] Handling call
0503 13:58:25.967629 (+1525581us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b540029fc54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:25.967736 (+ 107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1312149760}
I20260503 13:58:26.114293 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.114863 9445 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:26.115841 9566 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:26.126511 9566 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:26.126698 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.115994 9644 raft_consensus.cc:359] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.127643 9644 raft_consensus.cc:385] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:26.126523 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.128047 9644 raft_consensus.cc:740] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
W20260503 13:58:26.128937 9567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:26.128691 9644 consensus_queue.cc:260] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:26.129892 9492 leader_election.cc:343] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:26.130673 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:26.085937 10361 consensus_meta.cc:287] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.513s user 0.008s sys 0.002s
W20260503 13:58:26.053498 10332 consensus_meta.cc:287] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.529s user 0.008s sys 0.001s
I20260503 13:58:26.131312 9445 raft_consensus.cc:2393] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:26.128660 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:25.985070 9523 raft_consensus.cc:1240] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260503 13:58:26.137866 9523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 328) took 1635 ms. Trace:
I20260503 13:58:26.003782 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:25.991842 9600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 286) took 1407 ms. Trace:
I20260503 13:58:26.135641 10295 raft_consensus.cc:2804] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:26.138908 10295 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:26.139114 9522 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:25.945749 10039 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.140888 10039 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.125682 10319 raft_consensus.cc:2804] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:26.141494 10319 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:26.141815 10319 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:26.142030 10358 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:26.132053 10292 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:26.009029 10300 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.115141 9978 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260503 13:58:26.143179 9491 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:26.121793 10318 raft_consensus.cc:2804] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:26.125991 10342 leader_election.cc:290] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.009286 10193 consensus_queue.cc:1059] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.170s
I20260503 13:58:26.233530 9644 ts_tablet_manager.cc:1434] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.321s user 0.007s sys 0.000s
I20260503 13:58:26.089120 10303 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.002488 10315 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:25.949805 10204 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.173053 10346 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:26.232254 9594 raft_consensus.cc:2393] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:26.195643 10307 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.132162 10349 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.200122 10318 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:26.196059 10238 raft_consensus.cc:697] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:26.135476 10369 raft_consensus.cc:2804] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:26.135960 10372 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.252605 10315 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.253398 9642 tablet_bootstrap.cc:492] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:26.253823 9642 ts_tablet_manager.cc:1403] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.830s user 0.029s sys 0.007s
I20260503 13:58:26.254015 10372 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.242079 10193 consensus_queue.cc:1243] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Peer 1cb63886040047559d3893923cc9aaf6 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20260503 13:58:26.256630 10362 consensus_meta.cc:287] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.525s user 0.006s sys 0.003s
I20260503 13:58:26.246436 10339 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.178736 10332 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.088007 10363 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.137148 10361 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.196566 9978 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:26.235101 10295 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:26.138599 9600 rpcz_store.cc:276] 0503 13:58:24.584462 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:24.637462 (+ 53000us) service_pool.cc:224] Handling call
0503 13:58:25.059764 (+422302us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:25.328473 (+268709us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:25.990954 (+662481us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b5400261bd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:25.991648 (+ 694us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b5400261bd0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:25.991787 (+ 139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1499997440}
I20260503 13:58:26.349839 9596 raft_consensus.cc:1240] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:26.319648 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.138075 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.321631 9444 raft_consensus.cc:2393] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:26.132496 9447 raft_consensus.cc:2468] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:26.351305 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 267) took 1212 ms. Trace:
I20260503 13:58:26.268904 9602 raft_consensus.cc:2393] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:26.352118 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.326958 9521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 267) took 1820 ms. Trace:
W20260503 13:58:26.330775 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:26.353276 9447 raft_consensus.cc:2393] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:26.354435 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.355273 9447 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:26.138191 9523 rpcz_store.cc:276] 0503 13:58:24.502329 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:24.512753 (+ 10424us) service_pool.cc:224] Handling call
0503 13:58:25.333608 (+820855us) spinlock_profiling.cc:243] Waited 8.45 ms on lock 0x7b5800004598. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:25.984652 (+651044us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b5800004598. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea509ec 00007f9f3ea505ab 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:25.984836 (+ 184us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:26.137608 (+152772us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.137809 (+ 201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":116718720}
I20260503 13:58:26.338323 9528 raft_consensus.cc:2393] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:26.338335 9527 raft_consensus.cc:2393] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:26.338874 9524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 269) took 1831 ms. Trace:
I20260503 13:58:26.352151 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.357322 9524 rpcz_store.cc:276] 0503 13:58:24.506872 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.985882 (+1479010us) service_pool.cc:224] Handling call
0503 13:58:25.986187 (+ 305us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:25.986518 (+ 331us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:25.986546 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:25.986972 (+ 426us) log.cc:844] Serialized 26 byte log entry
0503 13:58:26.338136 (+351164us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:26.338264 (+ 128us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.338391 (+ 127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:26.338469 (+ 78us) raft_consensus.cc:1685] finished
0503 13:58:26.338493 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:26.338692 (+ 199us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":182,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":198820,"threads_started":1,"wal-append.queue_time_us":152980}
I20260503 13:58:26.359081 9450 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:26.359843 9566 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:26.360107 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.360677 9492 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:26.361269 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.353026 9521 rpcz_store.cc:276] 0503 13:58:24.506461 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.857485 (+1351024us) service_pool.cc:224] Handling call
0503 13:58:25.857844 (+ 359us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:25.862945 (+ 5101us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:25.862971 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:25.863325 (+ 354us) log.cc:844] Serialized 26 byte log entry
0503 13:58:25.865077 (+ 1752us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:25.865115 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:25.865178 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:26.326340 (+461162us) raft_consensus.cc:1685] finished
0503 13:58:26.326388 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:26.326728 (+ 340us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":75,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"thread_start_us":1469,"threads_started":1,"wal-append.queue_time_us":365001}
I20260503 13:58:26.361918 9450 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:26.362303 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:26.347056 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 268) took 1840 ms. Trace:
I20260503 13:58:26.325846 9520 raft_consensus.cc:1240] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:26.269810 9978 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:26.367762 9642 raft_consensus.cc:359] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.368799 9642 raft_consensus.cc:385] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:26.369095 9642 raft_consensus.cc:740] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:26.269378 10017 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.196293 10091 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.248953 10179 raft_consensus.cc:2804] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:26.143407 10287 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.379737 10091 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:26.255841 10289 consensus_meta.cc:287] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.586s user 0.005s sys 0.003s
I20260503 13:58:26.381307 10289 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.381868 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.382581 10332 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.382750 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 328) took 1293 ms. Trace:
I20260503 13:58:26.383167 9446 rpcz_store.cc:276] 0503 13:58:25.089753 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.093992 (+ 4239us) service_pool.cc:224] Handling call
0503 13:58:26.381360 (+1287368us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x7b5800000998. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46aff760 00007f9f46aff044 00007f9f4006c834 00007f9f4006c7c2
0503 13:58:26.382684 (+ 1324us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9487232}
I20260503 13:58:26.383734 10289 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.195519 10292 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:26.385063 9593 raft_consensus.cc:1240] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260503 13:58:26.258490 10297 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.389856 10315 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.232764 10300 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.390501 10315 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.392087 10300 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.250949 10307 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:26.392863 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:26.394897 10307 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.251267 10318 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:26.248575 10346 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.397496 10346 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.288237 10347 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:26.398361 10347 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
W20260503 13:58:26.395107 10340 consensus_meta.cc:287] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.660s user 0.004s sys 0.007s
I20260503 13:58:26.399420 10340 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.291646 10349 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.292006 10360 raft_consensus.cc:3055] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:26.401419 10360 raft_consensus.cc:740] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:26.403187 9643 tablet_bootstrap.cc:492] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:26.402698 10360 consensus_queue.cc:260] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.257270 10362 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.403872 9643 ts_tablet_manager.cc:1403] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 1.113s user 0.022s sys 0.006s
I20260503 13:58:26.404912 9452 raft_consensus.cc:2393] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 5: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:26.288205 10363 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.270537 10369 raft_consensus.cc:697] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:26.293582 10361 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.366647 10199 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.255820 10113 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.359874 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:26.371930 10193 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:26.357710 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:26.372144 10334 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.366521 10184 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.373317 10336 raft_consensus.cc:2804] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:26.373745 10238 consensus_queue.cc:237] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.375027 10204 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.387112 9597 raft_consensus.cc:2393] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:26.281227 10202 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.400734 10385 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:26.403879 10360 raft_consensus.cc:3060] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:26.392314 10315 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.405747 9642 consensus_queue.cc:260] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.413806 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:26.380399 10287 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.364010 9525 rpcz_store.cc:276] 0503 13:58:24.506667 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.922631 (+1415964us) service_pool.cc:224] Handling call
0503 13:58:26.105154 (+182523us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:26.105475 (+ 321us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:26.105502 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:26.105892 (+ 390us) log.cc:844] Serialized 26 byte log entry
0503 13:58:26.339006 (+233114us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:26.339079 (+ 73us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.339170 (+ 91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:26.346762 (+ 7592us) raft_consensus.cc:1685] finished
0503 13:58:26.346776 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:26.346918 (+ 142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":312,"raft.queue_time_us":1136,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":17150,"thread_start_us":200874,"threads_started":1,"wal-append.queue_time_us":155387}
I20260503 13:58:26.416033 10017 raft_consensus.cc:2804] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:26.415829 10184 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.416575 10336 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:26.416527 10361 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.423841 9522 raft_consensus.cc:2468] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:26.428037 10369 consensus_queue.cc:237] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.434556 10362 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.455698 10163 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.455770 10349 raft_consensus.cc:493] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.422595 9524 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:26.351711 9599 rpcz_store.cc:276] 0503 13:58:25.139199 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.923161 (+783962us) service_pool.cc:224] Handling call
0503 13:58:25.936437 (+ 13276us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:25.952627 (+ 16190us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:25.952657 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:25.953064 (+ 407us) log.cc:844] Serialized 26 byte log entry
0503 13:58:26.135440 (+182376us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:26.135478 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.135539 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:26.350792 (+215253us) raft_consensus.cc:1685] finished
0503 13:58:26.350822 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:26.351096 (+ 274us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15874,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":10409,"thread_start_us":30493,"threads_started":1,"wal-append.queue_time_us":289673}
I20260503 13:58:26.456923 9521 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:26.443114 9642 ts_tablet_manager.cc:1434] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.189s user 0.007s sys 0.006s
I20260503 13:58:26.461524 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:26.463178 10349 raft_consensus.cc:515] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.447935 9601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 258) took 1429 ms. Trace:
I20260503 13:58:26.464296 10224 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.449388 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:26.446260 10391 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.450747 10199 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.470703 9600 raft_consensus.cc:1240] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20260503 13:58:26.479918 10391 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.445217 10292 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:26.471102 10163 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.471699 10184 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.447041 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:26.450862 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:26.445482 9444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 313) took 1156 ms. Trace:
I20260503 13:58:26.451035 10193 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:26.443076 9595 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 236) took 3073 ms (client timeout 2185 ms). Trace:
I20260503 13:58:26.466917 9601 rpcz_store.cc:276] 0503 13:58:25.017904 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.293418 (+275514us) service_pool.cc:224] Handling call
0503 13:58:26.447554 (+1154136us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b5400061dd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:26.447738 (+ 184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1005774976}
I20260503 13:58:26.449954 9643 raft_consensus.cc:359] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.459903 10336 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:26.481772 10179 raft_consensus.cc:697] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:26.489358 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:26.489964 9602 raft_consensus.cc:2468] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
W20260503 13:58:26.490108 10304 consensus_meta.cc:287] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.559s user 0.009s sys 0.004s
I20260503 13:58:26.489372 10199 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.450170 10385 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.446102 10295 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.583225 9450 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:26.493845 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:26.536841 9491 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:26.466290 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 106776us
I20260503 13:58:26.594717 9643 raft_consensus.cc:385] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:26.493649 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:26.597690 10304 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.598407 10295 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.601588 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:26.596987 9643 raft_consensus.cc:740] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:26.492692 10017 raft_consensus.cc:697] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:26.491048 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:26.485281 10361 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.542702 10163 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.594695 10340 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.656276 9521 raft_consensus.cc:2468] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:26.617012 9644 tablet_bootstrap.cc:492] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:26.557235 9524 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:26.646270 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:26.604939 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 74178.5us
I20260503 13:58:26.596226 9599 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:26.492826 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.660118 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 261) took 1641 ms. Trace:
I20260503 13:58:26.598574 10318 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.660825 9597 rpcz_store.cc:276] 0503 13:58:25.018523 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.394948 (+376425us) service_pool.cc:224] Handling call
0503 13:58:26.659850 (+1264902us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x7b540031bad4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:26.660046 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2051155712}
I20260503 13:58:26.662292 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:26.654906 10391 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.663081 10303 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.663450 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:26.659045 9600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 268) took 1518 ms. Trace:
W20260503 13:58:26.663859 10358 consensus_meta.cc:287] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.521s user 0.010s sys 0.003s
I20260503 13:58:26.664510 9523 raft_consensus.cc:2468] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:26.664937 9445 raft_consensus.cc:2393] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
W20260503 13:58:26.665333 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 916362us
I20260503 13:58:26.665860 10318 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.666203 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 318) took 1301 ms. Trace:
I20260503 13:58:26.603411 9643 consensus_queue.cc:260] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.606724 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:26.605792 9642 tablet_bootstrap.cc:492] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:26.666688 9492 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:26.666582 9445 rpcz_store.cc:276] 0503 13:58:25.365136 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.144342 (+779206us) service_pool.cc:224] Handling call
0503 13:58:26.665991 (+521649us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b54002c06d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:26.666130 (+ 139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":722098944}
I20260503 13:58:26.664331 9597 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:26.667140 9600 rpcz_store.cc:276] 0503 13:58:25.140842 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.349621 (+1208779us) service_pool.cc:224] Handling call
0503 13:58:26.464621 (+115000us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:26.481617 (+ 16996us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.658783 (+177166us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54002bb1d0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:26.658973 (+ 190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":247266688}
I20260503 13:58:26.666261 9416 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:26.670974 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.671368 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:26.672009 9416 leader_election.cc:304] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:26.672452 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.669948 9525 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
W20260503 13:58:26.619093 9453 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 299) took 1334 ms (client timeout 1621 ms). Trace:
I20260503 13:58:26.673048 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:26.615099 9491 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:26.675405 9522 raft_consensus.cc:2468] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:26.677433 10318 raft_consensus.cc:2804] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:26.677884 10318 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:26.678272 10318 raft_consensus.cc:3060] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:26.678414 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:26.679764 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:26.673450 9453 rpcz_store.cc:269] 0503 13:58:25.284845 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.396291 (+111446us) service_pool.cc:224] Handling call
0503 13:58:26.618771 (+1222480us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b5400304654. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:26.618947 (+ 176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1207661952}
I20260503 13:58:26.614094 10193 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.681502 9450 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:26.675675 9520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 260) took 2170 ms. Trace:
I20260503 13:58:26.682330 10260 raft_consensus.cc:2804] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:26.545136 10319 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.606143 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:26.682226 9520 rpcz_store.cc:276] 0503 13:58:24.504930 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.694973 (+1190043us) service_pool.cc:224] Handling call
0503 13:58:25.695253 (+ 280us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:26.372047 (+676794us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.675425 (+303378us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b540038e354. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:26.675605 (+ 180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1448932224}
I20260503 13:58:26.684767 10295 raft_consensus.cc:493] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.604434 9593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 265) took 1478 ms. Trace:
I20260503 13:58:26.686946 9566 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:26.689064 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:26.486205 9444 rpcz_store.cc:276] 0503 13:58:25.288779 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.709695 (+420916us) service_pool.cc:224] Handling call
0503 13:58:26.445252 (+735557us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b54001d24d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:26.445414 (+ 162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1359522048}
I20260503 13:58:26.691270 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:26.692035 9567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:26.609990 9596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 266) took 1471 ms. Trace:
I20260503 13:58:26.677621 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.694386 10260 raft_consensus.cc:697] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:26.624106 9447 raft_consensus.cc:2468] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
W20260503 13:58:26.594817 9595 rpcz_store.cc:269] 0503 13:58:23.369507 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:23.657080 (+287573us) service_pool.cc:224] Handling call
0503 13:58:24.284793 (+627713us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b5800010898. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46aff760 00007f9f46aff044 00007f9f4006c834 00007f9f4006c7c2
0503 13:58:26.442666 (+2157873us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b5400058a54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:26.442900 (+ 234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1571013248}
I20260503 13:58:26.695818 9416 leader_election.cc:304] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:26.608680 10113 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.612525 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:26.681072 9600 raft_consensus.cc:2393] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:26.686612 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:26.716758 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 101886us
I20260503 13:58:26.717872 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:26.718386 9491 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:26.696260 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 96901.3us
I20260503 13:58:26.699329 9528 raft_consensus.cc:2393] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 3: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:26.605335 10304 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.700680 9521 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:26.701133 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.675737 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.699952 9593 rpcz_store.cc:276] 0503 13:58:25.126362 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.714852 (+588490us) service_pool.cc:224] Handling call
0503 13:58:25.727034 (+ 12182us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:26.385609 (+658575us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.604087 (+218478us) spinlock_profiling.cc:243] Waited 657 ms on lock 0x7b5400058f54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:26.604357 (+ 270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1511911808}
I20260503 13:58:26.702376 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.614593 9978 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.703663 9597 raft_consensus.cc:2393] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:26.703923 9451 raft_consensus.cc:2393] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:26.725649 9416 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:26.704519 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:26.704664 10260 consensus_queue.cc:237] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.701766 10295 raft_consensus.cc:515] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.700173 10358 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election won for term 5
W20260503 13:58:26.705782 10310 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s user 0.011s sys 0.022s
I20260503 13:58:26.624060 10347 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.706583 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:26.704470 10028 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:26.736940 10385 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.745853 10340 raft_consensus.cc:2804] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:26.690675 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:26.708489 9519 raft_consensus.cc:1275] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:26.680824 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 304) took 1387 ms. Trace:
I20260503 13:58:26.746246 10340 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:26.703471 9643 ts_tablet_manager.cc:1434] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.299s user 0.015s sys 0.009s
I20260503 13:58:26.714113 9525 raft_consensus.cc:2468] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:26.715248 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:26.748977 9445 rpcz_store.cc:276] 0503 13:58:25.286004 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.672210 (+1386206us) service_pool.cc:224] Handling call
0503 13:58:26.673090 (+ 880us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:26.720395 10303 raft_consensus.cc:2749] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:26.721436 10349 leader_election.cc:290] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.748616 9524 raft_consensus.cc:3060] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:26.750439 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.751220 10340 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:26.751508 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.752050 9445 raft_consensus.cc:3060] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:26.724387 9453 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:26.724620 10199 raft_consensus.cc:2804] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:26.725121 9599 raft_consensus.cc:2393] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 5: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:26.725322 9593 raft_consensus.cc:1240] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:26.719055 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:26.725792 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.723341 9492 leader_election.cc:304] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:26.728299 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:26.728482 10166 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:26.718487 10179 consensus_queue.cc:237] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.724808 9446 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:26.729169 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 335) took 1495 ms. Trace:
I20260503 13:58:26.729705 9452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 323) took 1149 ms. Trace:
I20260503 13:58:26.753245 10199 raft_consensus.cc:493] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:26.729156 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:26.756110 9448 rpcz_store.cc:276] 0503 13:58:25.233471 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.264726 (+ 31255us) service_pool.cc:224] Handling call
0503 13:58:26.728837 (+1464111us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b540039d5d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:26.728986 (+ 149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":380098944}
I20260503 13:58:26.756163 9452 rpcz_store.cc:276] 0503 13:58:25.579997 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.685596 (+105599us) service_pool.cc:224] Handling call
0503 13:58:26.729508 (+1043912us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b54002936d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:26.729650 (+ 142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1629440768}
I20260503 13:58:26.748699 9519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 257) took 2244 ms. Trace:
I20260503 13:58:26.757484 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.758249 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 305) took 1472 ms. Trace:
I20260503 13:58:26.731568 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 266) took 2225 ms. Trace:
I20260503 13:58:26.715854 10017 consensus_queue.cc:237] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.739301 9521 raft_consensus.cc:2468] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:26.736605 10193 raft_consensus.cc:2749] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:26.741043 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.742089 9566 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:26.743604 10358 raft_consensus.cc:697] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:26.744200 10295 leader_election.cc:290] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.744859 9520 raft_consensus.cc:2393] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
W20260503 13:58:26.743018 10310 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.802s user 0.011s sys 0.023s
I20260503 13:58:26.745246 10347 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.753193 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:26.770931 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 45443us
I20260503 13:58:26.753847 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:26.756662 9601 raft_consensus.cc:1275] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:26.757764 9519 rpcz_store.cc:276] 0503 13:58:24.504254 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.525916 (+1021662us) service_pool.cc:224] Handling call
0503 13:58:26.676388 (+1150472us) spinlock_profiling.cc:243] Waited 47.5 ms on lock 0x7b5800004598. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:26.707791 (+ 31403us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b5800004598. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea509ec 00007f9f3ea505ab 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:26.707988 (+ 197us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:26.748400 (+ 40412us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.748630 (+ 230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":119643264}
I20260503 13:58:26.756884 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:26.731274 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.734118 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:26.757778 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.703239 9596 rpcz_store.cc:276] 0503 13:58:25.138471 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.887452 (+748981us) service_pool.cc:224] Handling call
0503 13:58:25.887759 (+ 307us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:26.350401 (+462642us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.609733 (+259332us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b5400078854. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:26.609936 (+ 203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1060659584}
I20260503 13:58:26.760769 10199 raft_consensus.cc:3060] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:26.763469 10251 raft_consensus.cc:2749] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:26.763603 9449 raft_consensus.cc:2393] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:26.764504 9643 tablet_bootstrap.cc:492] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:26.764734 9595 raft_consensus.cc:1240] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260503 13:58:26.765548 9593 raft_consensus.cc:1240] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:26.766590 9492 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:26.759189 9526 rpcz_store.cc:276] 0503 13:58:24.506241 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.716697 (+1210456us) service_pool.cc:224] Handling call
0503 13:58:25.722045 (+ 5348us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:25.845319 (+123274us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.731040 (+885721us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54002aab54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:26.731355 (+ 315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":277709184}
I20260503 13:58:26.741761 9602 raft_consensus.cc:3060] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:26.758621 9448 rpcz_store.cc:276] 0503 13:58:25.286239 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.757170 (+1470931us) service_pool.cc:224] Handling call
0503 13:58:26.758202 (+ 1032us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:26.773743 10358 consensus_queue.cc:237] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.775040 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.755689 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.775705 9528 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:26.776583 9525 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:26.795435 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 299) took 1008 ms. Trace:
I20260503 13:58:26.757512 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:26.799536 9527 raft_consensus.cc:2468] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:26.778724 9418 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:26.788590 9567 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:26.801162 10243 raft_consensus.cc:2749] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:26.801717 9418 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:26.801976 9567 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:26.802748 10243 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:26.802572 10385 raft_consensus.cc:2804] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:26.784346 9601 raft_consensus.cc:1275] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:26.803226 10243 raft_consensus.cc:697] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:26.803534 10372 raft_consensus.cc:2804] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:26.805070 10372 raft_consensus.cc:697] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:26.785521 9595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 289) took 1020 ms. Trace:
I20260503 13:58:26.805351 10243 consensus_queue.cc:237] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.805562 9595 rpcz_store.cc:276] 0503 13:58:25.765251 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.724243 (+958992us) service_pool.cc:224] Handling call
0503 13:58:26.724736 (+ 493us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:26.784709 (+ 59973us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.785274 (+ 565us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b54003bf954. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:26.785465 (+ 191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":64136960}
I20260503 13:58:26.803663 10362 consensus_queue.cc:1059] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260503 13:58:26.793548 9599 raft_consensus.cc:1240] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:26.803013 9567 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:26.804390 9566 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:26.793208 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.796657 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.795781 9597 rpcz_store.cc:276] 0503 13:58:25.786634 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.745336 (+958702us) service_pool.cc:224] Handling call
0503 13:58:26.745623 (+ 287us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:26.745845 (+ 222us) raft_consensus.cc:1534] Early marking committed up to index 1
0503 13:58:26.752435 (+ 6590us) log.cc:844] Serialized 14 byte log entry
0503 13:58:26.795154 (+ 42719us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:26.795200 (+ 46us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:26.795234 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.795294 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:26.795390 (+ 96us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3202,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":72,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":29554,"threads_started":1}
I20260503 13:58:26.803840 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.808174 9451 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:26.810566 9445 raft_consensus.cc:2468] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:26.780272 9522 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:26.806857 10310 raft_consensus.cc:3055] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:26.834404 10310 raft_consensus.cc:740] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:26.837925 10304 raft_consensus.cc:2804] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:26.825145 10361 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.846289 10304 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:26.836958 9521 raft_consensus.cc:1275] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:58:26.838795 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.846700 10304 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:26.847008 9521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 270) took 2339 ms. Trace:
I20260503 13:58:26.846581 10310 consensus_queue.cc:260] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.847532 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.847332 9521 rpcz_store.cc:276] 0503 13:58:24.507117 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.777794 (+2270677us) service_pool.cc:224] Handling call
0503 13:58:26.836436 (+ 58642us) spinlock_profiling.cc:243] Waited 1.05 ms on lock 0x7b5800004598. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:26.836644 (+ 208us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:26.846778 (+ 10134us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.846939 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2411264}
I20260503 13:58:26.848194 9445 raft_consensus.cc:2468] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:26.824651 10113 raft_consensus.cc:2804] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:26.848857 10310 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:26.849087 10113 raft_consensus.cc:697] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:26.850025 10113 consensus_queue.cc:237] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.850610 9418 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:26.852308 9418 leader_election.cc:304] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:26.836252 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 48241.3us
I20260503 13:58:26.853288 9566 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:26.843004 9528 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:26.849056 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:26.853794 10300 raft_consensus.cc:2804] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:26.854172 9445 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:26.854291 10300 raft_consensus.cc:697] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:26.854437 9528 raft_consensus.cc:1240] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:26.855007 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 358) took 1099 ms. Trace:
I20260503 13:58:26.855365 9528 rpcz_store.cc:276] 0503 13:58:25.755367 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.854019 (+1098652us) service_pool.cc:224] Handling call
0503 13:58:26.854291 (+ 272us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:26.854787 (+ 496us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.854941 (+ 154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:26.855154 10300 consensus_queue.cc:237] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.853394 10362 raft_consensus.cc:2804] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:26.854950 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.856626 10362 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:26.857048 10362 raft_consensus.cc:3060] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:26.858340 9445 raft_consensus.cc:1275] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:26.858840 10347 raft_consensus.cc:2804] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:26.859294 10347 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:26.859637 10347 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:26.859887 10287 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.860330 10287 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.862107 10287 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:26.862670 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:26.807663 10163 raft_consensus.cc:2804] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:26.858876 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 296) took 1574 ms. Trace:
I20260503 13:58:26.864398 9445 rpcz_store.cc:276] 0503 13:58:25.283966 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.857671 (+1573705us) service_pool.cc:224] Handling call
0503 13:58:26.858017 (+ 346us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:26.858648 (+ 631us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.858801 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:26.865478 9450 raft_consensus.cc:2468] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:26.866107 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.804234 10385 raft_consensus.cc:697] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:26.836397 9594 raft_consensus.cc:1240] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:26.837316 10193 raft_consensus.cc:2749] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260503 13:58:26.863353 10287 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:26.838548 10319 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.868844 9452 raft_consensus.cc:2410] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:26.864413 9519 raft_consensus.cc:1240] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 3. Ops: []
I20260503 13:58:26.886023 9526 raft_consensus.cc:1240] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:26.865617 10163 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:26.872906 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 351) took 1074 ms. Trace:
W20260503 13:58:26.889334 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 121518us
I20260503 13:58:26.890290 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 344) took 1773 ms. Trace:
I20260503 13:58:26.890615 9526 rpcz_store.cc:276] 0503 13:58:25.116982 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.793211 (+1676229us) service_pool.cc:224] Handling call
0503 13:58:26.866959 (+ 73748us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:26.890067 (+ 23108us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.890232 (+ 165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:26.874346 9527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 279) took 1746 ms. Trace:
I20260503 13:58:26.894297 9491 leader_election.cc:304] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:26.881808 10310 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:26.805948 10372 consensus_queue.cc:237] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.895241 10310 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.897042 10310 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:26.897662 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:26.899103 9444 raft_consensus.cc:1240] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:26.882447 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:26.903206 9418 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:26.903298 9444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 300) took 1618 ms. Trace:
I20260503 13:58:26.884683 10295 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.886897 9596 raft_consensus.cc:1240] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:26.904615 9450 raft_consensus.cc:3060] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 2
I20260503 13:58:26.888080 10163 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:26.911775 9524 raft_consensus.cc:2468] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
W20260503 13:58:26.913407 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 60277.6us
I20260503 13:58:26.894017 9978 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.920928 10366 raft_consensus.cc:2804] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:26.921972 9566 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:26.923574 9453 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:26.924439 9451 raft_consensus.cc:1240] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:26.927964 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 301) took 1642 ms. Trace:
I20260503 13:58:26.899957 9452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 347) took 1188 ms. Trace:
I20260503 13:58:26.931361 9526 raft_consensus.cc:1240] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:26.928285 9451 rpcz_store.cc:276] 0503 13:58:25.285311 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.881221 (+1595910us) service_pool.cc:224] Handling call
0503 13:58:26.918703 (+ 37482us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:26.927750 (+ 9047us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.927911 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:26.905462 9519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 334) took 2355 ms. Trace:
I20260503 13:58:26.912832 9448 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:26.921588 10385 consensus_queue.cc:237] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.933568 10324 raft_consensus.cc:2749] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260503 13:58:26.933938 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:26.934100 10179 raft_consensus.cc:493] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.934921 9350 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51410 (request call id 64) took 7152 ms. Trace:
I20260503 13:58:26.934630 10179 raft_consensus.cc:515] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.912664 10251 raft_consensus.cc:2764] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260503 13:58:26.893029 10287 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.935247 9350 rpcz_store.cc:276] 0503 13:58:19.740858 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:19.749018 (+ 8160us) service_pool.cc:224] Handling call
0503 13:58:26.889139 (+7140121us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0503 13:58:23.068447 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 13:58:23.087616 (+ 19169us) write_op.cc:432] Acquiring schema lock in shared mode
0503 13:58:23.092623 (+ 5007us) write_op.cc:435] Acquired schema lock
0503 13:58:23.096408 (+ 3785us) tablet.cc:662] Decoding operations
0503 13:58:23.421453 (+325045us) write_op.cc:620] Acquiring the partition lock for write op
0503 13:58:23.447323 (+ 25870us) write_op.cc:641] Partition lock acquired for write op
0503 13:58:23.450515 (+ 3192us) tablet.cc:685] Acquiring locks for 6 operations
0503 13:58:23.609447 (+158932us) tablet.cc:701] Row locks acquired
0503 13:58:23.609467 (+ 20us) write_op.cc:260] PREPARE: finished
0503 13:58:23.609601 (+ 134us) write_op.cc:270] Start()
0503 13:58:23.609705 (+ 104us) write_op.cc:276] Timestamp: P: 1777816703609561 usec, L: 0
0503 13:58:23.609726 (+ 21us) op_driver.cc:348] REPLICATION: starting
0503 13:58:23.647885 (+ 38159us) log.cc:844] Serialized 2463 byte log entry
0503 13:58:24.517529 (+869644us) op_driver.cc:464] REPLICATION: finished
0503 13:58:24.906075 (+388546us) write_op.cc:301] APPLY: starting
0503 13:58:24.910703 (+ 4628us) tablet.cc:1367] starting BulkCheckPresence
0503 13:58:24.939718 (+ 29015us) tablet.cc:1370] finished BulkCheckPresence
0503 13:58:24.939848 (+ 130us) tablet.cc:1372] starting ApplyRowOperation cycle
0503 13:58:24.956851 (+ 17003us) tablet.cc:1383] finished ApplyRowOperation cycle
0503 13:58:24.957042 (+ 191us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0503 13:58:24.957089 (+ 47us) write_op.cc:312] APPLY: finished
0503 13:58:24.975292 (+ 18203us) log.cc:844] Serialized 52 byte log entry
0503 13:58:24.975461 (+ 169us) write_op.cc:489] Releasing partition, row and schema locks
0503 13:58:24.975585 (+ 124us) write_op.cc:454] Released schema lock
0503 13:58:24.981951 (+ 6366us) write_op.cc:341] FINISH: Updating metrics
0503 13:58:24.982824 (+ 873us) write_op.cc:489] Releasing partition, row and schema locks
0503 13:58:24.982855 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":390233,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":367816,"apply.run_cpu_time_us":2691,"apply.run_wall_time_us":81126,"num_ops":6,"prepare.queue_time_us":138483,"prepare.run_cpu_time_us":17565,"prepare.run_wall_time_us":891203,"raft.queue_time_us":365428,"raft.run_cpu_time_us":30620,"raft.run_wall_time_us":196184,"replication_time_us":899986,"thread_start_us":887721,"threads_started":4,"wal-append.queue_time_us":273758,"wal-append.run_cpu_time_us":98191,"wal-append.run_wall_time_us":2129274}]]}
W20260503 13:58:26.938048 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 59934.4us
I20260503 13:58:26.944649 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 297) took 1107 ms. Trace:
I20260503 13:58:26.892079 9447 rpcz_store.cc:276] 0503 13:58:25.798677 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.740380 (+941703us) service_pool.cc:224] Handling call
0503 13:58:26.868358 (+127978us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:26.944976 9526 rpcz_store.cc:276] 0503 13:58:25.837641 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.905672 (+1068031us) service_pool.cc:224] Handling call
0503 13:58:26.931210 (+ 25538us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:26.931696 (+ 486us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.944600 (+ 12904us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:26.934685 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:26.936213 10304 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.943158 9642 tablet_bootstrap.cc:654] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:26.943348 10362 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.947258 9418 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:26.891227 9527 rpcz_store.cc:276] 0503 13:58:25.117524 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.800202 (+1682678us) service_pool.cc:224] Handling call
0503 13:58:26.800525 (+ 323us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:26.837621 (+ 37096us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.863867 (+ 26246us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:26.957350 10170 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.958079 9451 raft_consensus.cc:1240] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:26.957835 10170 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.958613 10179 leader_election.cc:290] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.960028 10287 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:26.960767 9978 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:26.961285 9978 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.960096 10304 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.958659 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 312) took 1670 ms. Trace:
I20260503 13:58:26.962401 9451 rpcz_store.cc:276] 0503 13:58:25.288533 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.944363 (+1655830us) service_pool.cc:224] Handling call
0503 13:58:26.957863 (+ 13500us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:26.958427 (+ 564us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.958593 (+ 166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:26.962980 9978 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:26.954998 10193 raft_consensus.cc:2749] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:26.962738 10347 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.937150 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:27.001722 9491 leader_election.cc:343] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:27.002698 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 47193us
I20260503 13:58:27.002802 9491 leader_election.cc:304] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:26.908371 9444 rpcz_store.cc:276] 0503 13:58:25.285071 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.873126 (+1588055us) service_pool.cc:224] Handling call
0503 13:58:26.898938 (+ 25812us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:26.899476 (+ 538us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.903214 (+ 3738us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:26.966918 9418 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:26.917920 10295 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:26.925064 10318 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.003757 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:26.948026 9600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 294) took 1164 ms. Trace:
W20260503 13:58:26.965250 10402 log.cc:927] Time spent T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260503 13:58:26.930864 9452 rpcz_store.cc:276] 0503 13:58:25.704486 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.757511 (+1053025us) service_pool.cc:224] Handling call
0503 13:58:26.893140 (+135629us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:26.966704 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 52696.4us
I20260503 13:58:26.967796 9453 raft_consensus.cc:1240] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
W20260503 13:58:26.960732 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:26.952070 10319 consensus_queue.cc:1059] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.104s
I20260503 13:58:27.032496 9524 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:27.076282 9450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 297) took 1787 ms. Trace:
I20260503 13:58:26.967507 9445 raft_consensus.cc:1275] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260503 13:58:26.952453 10366 raft_consensus.cc:697] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:26.952778 10260 consensus_queue.cc:1059] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.113s
I20260503 13:58:27.005928 10184 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:27.096170 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 336) took 1784 ms. Trace:
I20260503 13:58:27.108856 9453 rpcz_store.cc:276] 0503 13:58:25.288646 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.938055 (+1649409us) service_pool.cc:224] Handling call
0503 13:58:26.967641 (+ 29586us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.072162 (+104521us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.072535 (+ 373us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:27.031435 10372 raft_consensus.cc:2804] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:27.099468 9445 pending_rounds.cc:85] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6: Aborting all ops after (but not including) 1
I20260503 13:58:26.957376 10310 consensus_queue.cc:1048] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.142s
I20260503 13:58:27.035629 10199 raft_consensus.cc:515] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:27.117638 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 164083us
I20260503 13:58:27.106544 9521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 356) took 1346 ms. Trace:
I20260503 13:58:27.109825 9445 pending_rounds.cc:107] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260503 13:58:27.121543 9445 raft_consensus.cc:2889] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:26.932768 9519 rpcz_store.cc:276] 0503 13:58:24.544944 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.777460 (+2232516us) service_pool.cc:224] Handling call
0503 13:58:26.824637 (+ 47177us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:26.887301 (+ 62664us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.900081 (+ 12780us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:27.114151 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:27.111554 9566 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:27.100365 10251 raft_consensus.cc:2749] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:27.123773 9491 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:27.100534 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:27.111449 10372 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:27.127465 9644 tablet_bootstrap.cc:654] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:27.109885 10184 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:27.057281 9600 rpcz_store.cc:276] 0503 13:58:25.779721 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.726141 (+946420us) service_pool.cc:224] Handling call
0503 13:58:26.726542 (+ 401us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:26.781831 (+ 55289us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:26.781869 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:26.782447 (+ 578us) log.cc:844] Serialized 26 byte log entry
0503 13:58:26.849360 (+ 66913us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:26.849400 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:26.849464 (+ 64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:26.924544 (+ 75080us) raft_consensus.cc:1685] finished
0503 13:58:26.924587 (+ 43us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:26.944406 (+ 19819us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1845,"raft.queue_time_us":5913,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"thread_start_us":66549,"threads_started":1,"wal-append.queue_time_us":104598}
I20260503 13:58:27.099844 9450 rpcz_store.cc:276] 0503 13:58:25.284307 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.874498 (+1590191us) service_pool.cc:224] Handling call
0503 13:58:26.904475 (+ 29977us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.071120 (+166645us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:27.071148 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:27.071188 (+ 40us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:27.071215 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.071271 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:27.071388 (+ 117us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":29958}
I20260503 13:58:27.155102 9446 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:27.156117 9524 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 6.
I20260503 13:58:27.123044 9447 raft_consensus.cc:1240] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:27.157569 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:27.144403 9566 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:27.127475 10372 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
W20260503 13:58:27.171195 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 51534us
I20260503 13:58:27.122830 9598 raft_consensus.cc:1275] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:58:27.130229 9521 rpcz_store.cc:276] 0503 13:58:25.755068 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.848281 (+1093213us) service_pool.cc:224] Handling call
0503 13:58:26.848621 (+ 340us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:26.848880 (+ 259us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:26.903934 (+ 55054us) log.cc:844] Serialized 14 byte log entry
0503 13:58:27.100003 (+196069us) log.cc:844] Serialized 14 byte log entry
0503 13:58:27.100994 (+ 991us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:27.101089 (+ 95us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:27.101167 (+ 78us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.101258 (+ 91us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:27.101366 (+ 108us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2558,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":4731,"thread_start_us":205264,"threads_started":2}
I20260503 13:58:27.108311 10238 consensus_queue.cc:1048] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
W20260503 13:58:27.168581 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 45182.9us
I20260503 13:58:27.144866 10385 raft_consensus.cc:3055] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:27.145202 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:27.177795 9566 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:27.150964 10324 raft_consensus.cc:2749] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:27.151265 9978 raft_consensus.cc:2804] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:27.124078 10340 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.183094 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 339) took 1711 ms. Trace:
I20260503 13:58:27.182889 9491 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:27.184725 9453 rpcz_store.cc:276] 0503 13:58:25.466323 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.114316 (+1647993us) service_pool.cc:224] Handling call
0503 13:58:27.165270 (+ 50954us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.177745 (+ 12475us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:27.177772 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:27.177817 (+ 45us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:27.177845 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.177901 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:27.177995 (+ 94us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1133440}
I20260503 13:58:27.180295 10385 raft_consensus.cc:740] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:27.157982 9448 raft_consensus.cc:2468] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:27.188017 9449 raft_consensus.cc:2393] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:27.188488 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:27.189186 9521 raft_consensus.cc:2393] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:27.173111 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 331) took 1026 ms. Trace:
I20260503 13:58:27.192131 9450 raft_consensus.cc:1240] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:27.192798 9450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 340) took 1707 ms. Trace:
I20260503 13:58:27.193102 9450 rpcz_store.cc:276] 0503 13:58:25.485483 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.168764 (+1683281us) service_pool.cc:224] Handling call
0503 13:58:27.184634 (+ 15870us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.192584 (+ 7950us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.192749 (+ 165us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:27.193761 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 146016us
I20260503 13:58:27.191272 10253 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:27.190114 9566 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:27.195528 9567 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:27.180536 10318 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:27.167542 10184 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:27.196679 9418 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:27.189805 10411 log.cc:927] Time spent T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20260503 13:58:27.205281 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 308) took 1914 ms. Trace:
I20260503 13:58:27.205766 9445 rpcz_store.cc:276] 0503 13:58:25.287409 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.886735 (+1599326us) service_pool.cc:224] Handling call
0503 13:58:26.924445 (+ 37710us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.164499 (+240054us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.200956 (+ 36457us) spinlock_profiling.cc:243] Waited 42.7 ms on lock 0x7b540007fdd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:27.201116 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":98263424}
I20260503 13:58:27.186720 9978 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:27.186872 10362 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:27.188622 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 316) took 1864 ms. Trace:
I20260503 13:58:27.199816 10358 consensus_queue.cc:1048] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260503 13:58:27.189510 10347 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:27.192991 10319 raft_consensus.cc:2749] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:27.209683 9566 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:27.225908 9566 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:27.214756 9491 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:27.215394 9416 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:27.221638 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:27.199397 10251 raft_consensus.cc:2749] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260503 13:58:27.213418 10336 consensus_meta.cc:287] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.615s user 0.008s sys 0.002s
I20260503 13:58:27.202787 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:27.216008 9448 raft_consensus.cc:1240] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:27.250239 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 325) took 1545 ms. Trace:
I20260503 13:58:27.250590 9448 rpcz_store.cc:276] 0503 13:58:25.704825 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.215421 (+1510596us) service_pool.cc:224] Handling call
0503 13:58:27.215821 (+ 400us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.250005 (+ 34184us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.250183 (+ 178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:27.251978 9448 raft_consensus.cc:1240] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:27.250407 9526 raft_consensus.cc:2468] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
W20260503 13:58:27.216986 10412 log.cc:927] Time spent T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20260503 13:58:27.216688 10283 raft_consensus.cc:2804] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:27.217124 10163 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.216336 10396 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:27.222137 10017 raft_consensus.cc:2804] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:27.192131 9446 rpcz_store.cc:276] 0503 13:58:26.144886 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.675403 (+530517us) service_pool.cc:224] Handling call
0503 13:58:27.171337 (+495934us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:27.208647 9978 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:27.238180 10170 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:27.238355 10419 raft_consensus.cc:2804] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:27.239128 9522 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:27.239329 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 354) took 1278 ms. Trace:
I20260503 13:58:27.238135 9524 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:27.239985 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:27.240628 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 295) took 1478 ms. Trace:
I20260503 13:58:27.240823 9643 tablet_bootstrap.cc:654] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:27.258109 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:27.217384 10385 consensus_queue.cc:260] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.202555 10366 consensus_queue.cc:237] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.242321 10340 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:27.228436 10318 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:27.222597 10243 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election won for term 6
I20260503 13:58:27.204739 10253 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.245487 10184 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.231252 9566 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:27.236384 9450 raft_consensus.cc:1240] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:27.214255 10199 leader_election.cc:290] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:27.267374 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:27.217434 9452 raft_consensus.cc:1240] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:27.257938 9445 rpcz_store.cc:276] 0503 13:58:25.961277 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.218115 (+1256838us) service_pool.cc:224] Handling call
0503 13:58:27.238860 (+ 20745us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.239026 (+ 166us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:27.239052 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:27.239092 (+ 40us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:27.239120 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.239166 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:27.239266 (+ 100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:27.258136 9528 rpcz_store.cc:276] 0503 13:58:25.762275 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.872713 (+1110438us) service_pool.cc:224] Handling call
0503 13:58:26.888695 (+ 15982us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:26.909189 (+ 20494us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:26.909259 (+ 70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:26.926169 (+ 16910us) log.cc:844] Serialized 26 byte log entry
0503 13:58:27.101432 (+175263us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:27.101465 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.101513 (+ 48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:27.240399 (+138886us) raft_consensus.cc:1685] finished
0503 13:58:27.240422 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:27.240542 (+ 120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4086,"raft.queue_time_us":8779,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":10202,"thread_start_us":145227,"threads_started":1,"wal-append.queue_time_us":256517}
I20260503 13:58:27.253677 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:27.301628 9526 raft_consensus.cc:2468] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:27.230624 10193 raft_consensus.cc:2804] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260503 13:58:27.273185 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:27.274801 9450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 346) took 1585 ms. Trace:
I20260503 13:58:27.296824 9453 raft_consensus.cc:1240] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:27.256637 9525 raft_consensus.cc:1240] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5. Ops: []
W20260503 13:58:27.297448 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 98006.1us
I20260503 13:58:27.303356 9450 rpcz_store.cc:276] 0503 13:58:25.689300 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.193935 (+1504635us) service_pool.cc:224] Handling call
0503 13:58:27.236214 (+ 42279us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.274565 (+ 38351us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.274746 (+ 181us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:27.255741 10401 log.cc:927] Time spent T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.314s user 0.003s sys 0.001s
I20260503 13:58:27.256448 10028 raft_consensus.cc:2764] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260503 13:58:27.252540 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 330) took 1199 ms. Trace:
I20260503 13:58:27.217448 9447 rpcz_store.cc:276] 0503 13:58:25.322895 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.002417 (+1679522us) service_pool.cc:224] Handling call
0503 13:58:27.107604 (+105187us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.172552 (+ 64948us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.186826 (+ 14274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:27.253296 10283 raft_consensus.cc:697] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:27.223763 10324 consensus_queue.cc:1059] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.046s
I20260503 13:58:27.254796 10163 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:27.323820 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:27.255888 10017 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:27.301551 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:27.384713 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 153517us
I20260503 13:58:27.296367 10318 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:27.388907 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 174284us
I20260503 13:58:27.237831 10336 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.422257 9491 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:27.384505 10415 log.cc:927] Time spent T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20260503 13:58:27.392174 9449 raft_consensus.cc:1240] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:27.248454 10372 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.389909 10028 consensus_queue.cc:1059] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260503 13:58:27.393200 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.362161 10193 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:27.393446 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:27.433907 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.434597 9594 raft_consensus.cc:2393] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:27.435339 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.435842 9594 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:27.394124 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.394915 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.395095 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.386736 9448 rpcz_store.cc:276] 0503 13:58:26.053137 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.251388 (+1198251us) service_pool.cc:224] Handling call
0503 13:58:27.251800 (+ 412us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.252334 (+ 534us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.252494 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:27.302947 10347 raft_consensus.cc:493] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:27.396893 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:27.398445 10248 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:27.399730 10243 raft_consensus.cc:697] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 6 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:27.400153 10340 consensus_queue.cc:1059] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:27.393123 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:27.302175 10385 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:27.265420 10358 consensus_queue.cc:1048] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.121s
I20260503 13:58:27.254949 10251 consensus_queue.cc:1059] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.131s
I20260503 13:58:27.423375 9566 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:27.423677 9597 raft_consensus.cc:1240] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260503 13:58:27.387856 10304 raft_consensus.cc:2749] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:27.309285 9602 raft_consensus.cc:2468] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:27.255102 10396 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.427331 9449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 336) took 1094 ms. Trace:
I20260503 13:58:27.388345 10362 raft_consensus.cc:2749] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:27.299422 10419 raft_consensus.cc:697] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:27.363179 9522 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:27.403151 10420 consensus_queue.cc:1059] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20260503 13:58:27.432204 10193 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:27.465206 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:27.432807 9642 tablet_bootstrap.cc:492] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:27.432689 9595 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:27.396382 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:27.472409 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 216995us
I20260503 13:58:27.422989 10017 raft_consensus.cc:3060] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:27.449911 9523 raft_consensus.cc:2393] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:27.459589 10304 raft_consensus.cc:2804] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:27.462268 10199 raft_consensus.cc:3055] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:27.455404 9644 tablet_bootstrap.cc:492] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:27.455446 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 345) took 2338 ms. Trace:
I20260503 13:58:27.499605 9644 ts_tablet_manager.cc:1403] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.883s user 0.030s sys 0.030s
I20260503 13:58:27.499781 9525 rpcz_store.cc:276] 0503 13:58:25.117261 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.809543 (+1692282us) service_pool.cc:224] Handling call
0503 13:58:26.880170 (+ 70627us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.303683 (+423513us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.455250 (+151567us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b54000c58d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:27.455397 (+ 147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":865322752}
I20260503 13:58:27.458539 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 345) took 1889 ms. Trace:
I20260503 13:58:27.500748 9453 rpcz_store.cc:276] 0503 13:58:25.569563 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.188214 (+1618651us) service_pool.cc:224] Handling call
0503 13:58:27.191704 (+ 3490us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.303469 (+111765us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.455939 (+152470us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54001d3654. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:27.458481 (+ 2542us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":241360128}
I20260503 13:58:27.459122 9566 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:27.445012 9446 raft_consensus.cc:1275] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:27.445173 10163 raft_consensus.cc:3055] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Stepping down as leader of term 4
W20260503 13:58:27.462436 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 265806us
I20260503 13:58:27.449370 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:27.470623 9642 ts_tablet_manager.cc:1403] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.865s user 0.025s sys 0.031s
I20260503 13:58:27.470731 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:27.456231 10179 raft_consensus.cc:2764] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260503 13:58:27.451848 9449 rpcz_store.cc:276] 0503 13:58:26.333024 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.300776 (+967752us) service_pool.cc:224] Handling call
0503 13:58:27.392064 (+ 91288us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.427093 (+ 35029us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.427285 (+ 192us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:27.496986 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:27.445303 10347 raft_consensus.cc:515] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.472738 9600 raft_consensus.cc:3060] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:27.446184 10248 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.471652 10184 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:27.509883 9418 leader_election.cc:304] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:27.506438 10419 consensus_queue.cc:237] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.522929 10248 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:27.515305 10428 raft_consensus.cc:3055] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:27.523851 10428 raft_consensus.cc:740] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:27.523981 9593 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:27.524663 10428 consensus_queue.cc:260] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.525573 10428 raft_consensus.cc:3060] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:27.506736 10304 raft_consensus.cc:697] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:27.515594 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.529040 9596 raft_consensus.cc:2393] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:27.529178 10304 consensus_queue.cc:237] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.529848 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.530341 9596 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:27.523872 9644 raft_consensus.cc:359] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.533241 9644 raft_consensus.cc:385] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:27.533473 9644 raft_consensus.cc:740] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:27.534031 9642 raft_consensus.cc:359] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.534627 9642 raft_consensus.cc:385] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:27.534870 9642 raft_consensus.cc:740] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:27.535432 9642 consensus_queue.cc:260] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.498862 10283 consensus_queue.cc:237] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.542728 10428 raft_consensus.cc:3055] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:27.543102 10428 raft_consensus.cc:740] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:27.543819 10428 consensus_queue.cc:260] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.503839 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:27.545511 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:27.545635 9524 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:27.540510 9596 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:27.546844 9524 raft_consensus.cc:1240] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:27.520287 10113 raft_consensus.cc:2749] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:27.506522 10163 raft_consensus.cc:740] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:27.552676 10243 consensus_queue.cc:237] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.553417 10163 consensus_queue.cc:260] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.554683 10163 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
W20260503 13:58:27.555615 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 45857.6us
I20260503 13:58:27.516573 9519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 365) took 1169 ms. Trace:
I20260503 13:58:27.556841 9519 rpcz_store.cc:276] 0503 13:58:26.342988 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.421918 (+1078930us) service_pool.cc:224] Handling call
0503 13:58:27.453516 (+ 31598us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.469876 (+ 16360us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:27.474038 (+ 4162us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:27.496857 (+ 22819us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:27.499476 (+ 2619us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.504833 (+ 5357us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:27.512148 (+ 7315us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:27.558370 9519 raft_consensus.cc:1275] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:27.499341 10199 raft_consensus.cc:740] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:27.506340 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 335) took 1173 ms. Trace:
I20260503 13:58:27.504910 10170 raft_consensus.cc:3055] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:27.560699 9525 raft_consensus.cc:1240] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:27.559674 10358 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:27.567029 10428 raft_consensus.cc:3060] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:27.513176 10251 raft_consensus.cc:2749] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:27.571832 9601 raft_consensus.cc:3060] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:27.572104 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.573880 9642 ts_tablet_manager.cc:1434] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.066s user 0.005s sys 0.000s
W20260503 13:58:27.587286 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 128237us
W20260503 13:58:27.588472 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 166282us
I20260503 13:58:27.548113 9524 raft_consensus.cc:1240] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:27.573915 10358 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.593062 9492 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:27.569306 10170 raft_consensus.cc:740] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:27.604678 9566 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:27.605540 9521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 305) took 1178 ms. Trace:
I20260503 13:58:27.605865 9521 rpcz_store.cc:276] 0503 13:58:26.427340 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.425718 (+998378us) service_pool.cc:224] Handling call
0503 13:58:27.442734 (+ 17016us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.442912 (+ 178us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:27.450455 (+ 7543us) log.cc:844] Serialized 14 byte log entry
0503 13:58:27.605224 (+154769us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:27.605298 (+ 74us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:27.605331 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.605388 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:27.605486 (+ 98us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11784,"prepare.queue_time_us":7612,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":2930,"thread_start_us":128383,"threads_started":1}
I20260503 13:58:27.569052 10425 raft_consensus.cc:2804] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:27.609400 9599 raft_consensus.cc:1240] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:27.616302 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:27.620087 9521 raft_consensus.cc:2393] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
W20260503 13:58:27.617045 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 50066.3us
I20260503 13:58:27.625875 9593 raft_consensus.cc:2468] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:27.626652 10372 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:27.629475 9644 consensus_queue.cc:260] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.630980 9644 ts_tablet_manager.cc:1434] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.131s user 0.007s sys 0.007s
I20260503 13:58:27.631882 9644 tablet_bootstrap.cc:492] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:27.634111 9520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 341) took 2657 ms. Trace:
I20260503 13:58:27.635537 9643 tablet_bootstrap.cc:492] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:27.634492 9520 rpcz_store.cc:276] 0503 13:58:24.977050 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.777588 (+1800538us) service_pool.cc:224] Handling call
0503 13:58:26.797325 (+ 19737us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:26.837991 (+ 40666us) raft_consensus.cc:1534] Early marking committed up to index 1
0503 13:58:26.845015 (+ 7024us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:26.910685 (+ 65670us) log.cc:844] Serialized 26 byte log entry
0503 13:58:27.055352 (+144667us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:27.063755 (+ 8403us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.067539 (+ 3784us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:27.235496 (+167957us) raft_consensus.cc:1685] finished
0503 13:58:27.235518 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:27.633848 (+398330us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b54002067d0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:27.634053 (+ 205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9536,"raft.queue_time_us":35885,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":10719,"spinlock_wait_cycles":32050048,"thread_start_us":95871,"threads_started":1,"wal-append.queue_time_us":322748}
I20260503 13:58:27.635984 9643 ts_tablet_manager.cc:1403] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.872s user 0.036s sys 0.024s
I20260503 13:58:27.582623 10184 consensus_queue.cc:1059] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260503 13:58:27.638293 9491 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:27.605424 10318 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:27.654739 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 252) took 2638 ms. Trace:
I20260503 13:58:27.655485 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:27.610591 10430 raft_consensus.cc:2749] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:27.639587 10166 raft_consensus.cc:3055] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:27.567226 9446 rpcz_store.cc:276] 0503 13:58:26.332641 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.307596 (+974955us) service_pool.cc:224] Handling call
0503 13:58:27.419036 (+111440us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.506069 (+ 87033us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.506287 (+ 218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:27.658154 9599 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:27.610721 10336 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:27.657392 9598 rpcz_store.cc:276] 0503 13:58:25.016432 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:25.018691 (+ 2259us) service_pool.cc:224] Handling call
0503 13:58:25.019030 (+ 339us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.175718 (+2156688us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.654474 (+478756us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x7b5400077e54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:27.654672 (+ 198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4699418368}
I20260503 13:58:27.622049 10361 raft_consensus.cc:2804] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:27.623016 10425 raft_consensus.cc:697] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:27.688300 9521 raft_consensus.cc:1275] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260503 13:58:27.687542 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 131316us
I20260503 13:58:27.693171 10361 raft_consensus.cc:697] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:27.678507 10166 raft_consensus.cc:740] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:27.698837 10361 consensus_queue.cc:237] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.698997 10166 consensus_queue.cc:260] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.699987 10166 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:27.619709 10199 consensus_queue.cc:260] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.703403 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 338) took 2247 ms. Trace:
I20260503 13:58:27.703751 9451 rpcz_store.cc:276] 0503 13:58:25.455476 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.963650 (+1508174us) service_pool.cc:224] Handling call
0503 13:58:26.963982 (+ 332us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:26.964321 (+ 339us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:26.964345 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:26.964835 (+ 490us) log.cc:844] Serialized 26 byte log entry
0503 13:58:27.103169 (+138334us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:27.130982 (+ 27813us) log.cc:844] Serialized 14 byte log entry
0503 13:58:27.131386 (+ 404us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.131473 (+ 87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:27.631398 (+499925us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b1800032380. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea8db62 00007f9f3ea8fe01 00007f9f3ea8fdb9 00007f9f3ea8fd49 00007f9f3ea8fd11 00007f9f3ea8f00d 0000000000379254 0000000000379099 00007f9f3ea70fba
0503 13:58:27.701086 (+ 69688us) spinlock_profiling.cc:243] Waited 1.03 ms on lock 0x7b1800032380. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea8da33 00007f9f3ea8fe01 00007f9f3ea8fdb9 00007f9f3ea8fd49 00007f9f3ea8fd11 00007f9f3ea8f00d 0000000000379254 0000000000379099 00007f9f3ea70fba
0503 13:58:27.703193 (+ 2107us) raft_consensus.cc:1685] finished
0503 13:58:27.703215 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:27.703330 (+ 115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12092,"prepare.queue_time_us":38229,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":1904,"raft.queue_time_us":17033,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":74955,"spinlock_wait_cycles":5187840,"thread_start_us":134126,"threads_started":1,"wal-append.queue_time_us":202866}
I20260503 13:58:27.705907 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:27.705943 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:27.709862 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 90102.2us
I20260503 13:58:27.718436 9491 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:27.719058 10424 consensus_queue.cc:1048] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260503 13:58:27.720120 9491 leader_election.cc:343] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:27.720481 9491 leader_election.cc:304] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:27.721203 10336 raft_consensus.cc:2749] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:27.721961 9491 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:27.722648 10336 raft_consensus.cc:2749] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:27.723486 9491 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:27.727447 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 312) took 1014 ms. Trace:
I20260503 13:58:27.708129 9520 raft_consensus.cc:1275] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:27.727797 9525 rpcz_store.cc:276] 0503 13:58:26.713292 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.505947 (+792655us) service_pool.cc:224] Handling call
0503 13:58:27.560519 (+ 54572us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.567722 (+ 7203us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.727283 (+159561us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b540004e550. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:27.727395 (+ 112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":99028224}
I20260503 13:58:27.689298 10251 consensus_queue.cc:1059] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.028s
I20260503 13:58:27.708966 10295 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:27.686005 10170 consensus_queue.cc:260] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.757221 10310 raft_consensus.cc:493] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:27.773123 10425 consensus_queue.cc:237] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.773675 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:27.773475 10310 raft_consensus.cc:515] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.774724 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:27.775408 9520 raft_consensus.cc:2393] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:27.776108 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:27.776429 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 50838.7us
I20260503 13:58:27.772897 9599 raft_consensus.cc:1275] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:27.777141 10170 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:27.777472 9566 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:27.777493 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.779466 10283 raft_consensus.cc:2804] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:27.779937 10283 raft_consensus.cc:697] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:27.780124 10243 consensus_queue.cc:1059] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:58:27.780727 10283 consensus_queue.cc:237] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.781040 10243 consensus_queue.cc:1059] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:27.781780 10243 raft_consensus.cc:3055] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:27.781852 10113 raft_consensus.cc:3055] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:27.782284 10113 raft_consensus.cc:740] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:27.783077 10113 consensus_queue.cc:260] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.729746 9594 raft_consensus.cc:2468] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:27.784215 10113 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:27.710780 10396 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:27.766582 9643 raft_consensus.cc:359] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.788410 9643 raft_consensus.cc:385] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:27.788666 9643 raft_consensus.cc:740] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:27.758100 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.789187 9598 raft_consensus.cc:3060] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:27.790633 9643 consensus_queue.cc:260] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:27.796993 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:27.796669 9601 raft_consensus.cc:2468] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:27.784900 10310 leader_election.cc:290] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:27.776188 10426 log.cc:927] Time spent T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
W20260503 13:58:27.754549 10437 log.cc:927] Time spent T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
W20260503 13:58:27.798669 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 75204.9us
I20260503 13:58:27.753880 10427 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:27.754858 9524 raft_consensus.cc:1275] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260503 13:58:27.802304 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 307) took 1207 ms. Trace:
I20260503 13:58:27.792641 9520 raft_consensus.cc:3060] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:27.787644 10347 leader_election.cc:290] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:27.804471 9526 rpcz_store.cc:276] 0503 13:58:26.595405 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.444460 (+849055us) service_pool.cc:224] Handling call
0503 13:58:27.458325 (+ 13865us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.471460 (+ 13135us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:27.709559 (+238099us) log.cc:844] Serialized 14 byte log entry
0503 13:58:27.802010 (+ 92451us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:27.802057 (+ 47us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:27.802091 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.802146 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:27.802244 (+ 98us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5838,"prepare.queue_time_us":31051,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":40686,"thread_start_us":284883,"threads_started":2}
I20260503 13:58:27.795580 10336 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:27.804908 9491 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:27.730950 9595 raft_consensus.cc:2468] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:27.806749 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:27.791710 10243 raft_consensus.cc:740] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:27.810863 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:27.810563 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:27.811968 9451 raft_consensus.cc:2468] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:27.816875 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:27.817941 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:27.818403 9525 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:27.819403 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:27.820765 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:27.821835 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:27.822518 9524 raft_consensus.cc:2410] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:27.823330 9524 raft_consensus.cc:1240] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:27.807642 10358 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:27.812891 10336 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.826035 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.803747 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.826889 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:27.827443 9525 raft_consensus.cc:2468] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:27.830307 9598 raft_consensus.cc:2468] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:27.826133 10336 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:27.827611 9978 consensus_meta.cc:287] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.528s user 0.009s sys 0.000s
I20260503 13:58:27.838471 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:27.810637 10432 raft_consensus.cc:2764] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260503 13:58:27.842211 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:27.835028 9522 raft_consensus.cc:2410] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:27.799098 10017 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:27.835572 10421 log.cc:927] Time spent T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.595s user 0.010s sys 0.003s
I20260503 13:58:27.854351 9595 raft_consensus.cc:2393] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:27.852532 9978 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.854718 10193 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.883848 9644 tablet_bootstrap.cc:654] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:27.885048 10193 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:27.855546 9520 raft_consensus.cc:2468] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:27.863881 9566 leader_election.cc:304] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
W20260503 13:58:27.870142 10406 log.cc:927] Time spent T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.732s user 0.012s sys 0.000s
I20260503 13:58:27.871872 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:27.872251 9445 raft_consensus.cc:1240] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260503 13:58:27.873386 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:27.874019 9451 raft_consensus.cc:2368] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3.
I20260503 13:58:27.876713 9525 raft_consensus.cc:1275] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:27.891058 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 341) took 1168 ms. Trace:
I20260503 13:58:27.878891 9450 raft_consensus.cc:1240] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:27.880298 9600 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:27.873132 9594 raft_consensus.cc:3060] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:27.879503 10396 raft_consensus.cc:493] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:27.882488 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:27.891904 10396 raft_consensus.cc:515] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.892515 9522 raft_consensus.cc:2410] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:27.883478 9598 raft_consensus.cc:1240] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:27.891376 9445 rpcz_store.cc:276] 0503 13:58:26.722281 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.301276 (+578995us) service_pool.cc:224] Handling call
0503 13:58:27.871864 (+570588us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b5800000998. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:27.872045 (+ 181us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:27.890838 (+ 18793us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.891010 (+ 172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5415168}
I20260503 13:58:27.893950 10396 leader_election.cc:290] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:27.894096 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:27.895117 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:27.896117 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:27.897516 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:27.897686 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:27.898449 9450 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:27.899183 9594 raft_consensus.cc:2468] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:27.886605 9643 ts_tablet_manager.cc:1434] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.250s user 0.009s sys 0.000s
I20260503 13:58:27.900439 9643 tablet_bootstrap.cc:492] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:27.903998 9450 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:27.905339 9445 raft_consensus.cc:2393] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:27.906126 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:27.886921 10434 raft_consensus.cc:2749] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:27.907111 9524 raft_consensus.cc:3055] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:27.909514 9597 raft_consensus.cc:2393] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:27.906318 9643 tablet_bootstrap.cc:654] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:27.891351 10193 raft_consensus.cc:2804] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260503 13:58:27.921008 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 116178us
I20260503 13:58:27.935222 9451 raft_consensus.cc:1240] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:27.933790 9566 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:27.941561 9524 raft_consensus.cc:740] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:27.947352 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:27.934459 9445 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:27.949054 9491 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:27.952075 9566 leader_election.cc:343] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:27.952239 9567 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:27.916837 9448 raft_consensus.cc:2393] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:27.952982 9567 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:27.954023 10166 raft_consensus.cc:2749] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:27.955379 9526 raft_consensus.cc:1275] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260503 13:58:27.955892 9567 leader_election.cc:343] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:27.955286 9418 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:27.956215 9567 leader_election.cc:304] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:27.957541 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:27.958179 9525 raft_consensus.cc:2468] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:27.912098 10396 raft_consensus.cc:493] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:27.911897 10427 raft_consensus.cc:2749] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:27.943362 10430 consensus_queue.cc:1059] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20260503 13:58:27.951818 9524 consensus_queue.cc:260] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.961256 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.962069 9447 raft_consensus.cc:1275] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:58:27.945569 10193 raft_consensus.cc:493] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:27.959401 10432 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:27.959470 10251 raft_consensus.cc:2749] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:27.965189 9978 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:27.965335 9566 leader_election.cc:400] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Vote denied by peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599) with higher term. Message: Invalid argument: T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3.
I20260503 13:58:27.965108 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:27.912827 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:27.973430 9521 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:27.967968 9491 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:27.975226 9524 raft_consensus.cc:3060] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:27.974548 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:27.951860 10113 raft_consensus.cc:2804] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:27.960479 10347 raft_consensus.cc:2804] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:27.980335 9567 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:27.982733 10435 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:27.982978 10445 consensus_queue.cc:1048] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:27.982187 10430 consensus_queue.cc:1243] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer 1cb63886040047559d3893923cc9aaf6 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260503 13:58:27.984607 9418 leader_election.cc:343] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:27.986068 10170 consensus_queue.cc:1048] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:27.987248 10170 consensus_queue.cc:1059] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:58:27.987964 10366 raft_consensus.cc:3055] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:27.988277 10366 raft_consensus.cc:740] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:27.988852 10248 consensus_queue.cc:1059] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:27.976667 10310 raft_consensus.cc:2804] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:27.981325 10441 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:27.981720 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:27.988986 10366 consensus_queue.cc:260] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:27.989712 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:27.970202 10166 raft_consensus.cc:2749] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:27.990175 10366 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:27.990343 9595 raft_consensus.cc:1275] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260503 13:58:27.991302 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:27.992666 9594 raft_consensus.cc:1275] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:27.993238 10310 raft_consensus.cc:697] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:27.990250 9521 raft_consensus.cc:1240] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260503 13:58:27.972038 10425 raft_consensus.cc:493] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:27.984476 10405 log.cc:927] Time spent T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.799s user 0.000s sys 0.004s
I20260503 13:58:27.998443 9566 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:27.974946 10193 raft_consensus.cc:3060] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:27.993621 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:27.992353 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:28.008440 9452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 321) took 2543 ms. Trace:
I20260503 13:58:28.008843 9452 rpcz_store.cc:276] 0503 13:58:25.465276 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.102896 (+1637620us) service_pool.cc:224] Handling call
0503 13:58:27.192600 (+ 89704us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.385763 (+193163us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:28.008240 (+622477us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b540017aa50. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:28.008382 (+ 142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54202624}
I20260503 13:58:28.010687 9416 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:28.011411 10260 raft_consensus.cc:2749] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260503 13:58:28.011709 9416 leader_election.cc:343] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:28.012105 9416 leader_election.cc:304] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:27.945778 10243 consensus_queue.cc:260] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.015928 9978 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:27.989014 10017 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:27.971293 10396 raft_consensus.cc:515] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.017025 9418 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:27.995882 10361 raft_consensus.cc:3060] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:28.017161 9524 raft_consensus.cc:1275] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
W20260503 13:58:28.000795 9494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:27.996783 10347 raft_consensus.cc:697] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:28.028290 9978 raft_consensus.cc:697] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:27.980911 9446 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:27.994217 10113 raft_consensus.cc:697] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:28.032403 9418 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:27.999163 9519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 316) took 1050 ms. Trace:
I20260503 13:58:28.034499 9566 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:28.034432 9416 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:28.029644 9524 pending_rounds.cc:85] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a: Aborting all ops after (but not including) 0
I20260503 13:58:28.043624 9524 pending_rounds.cc:107] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260503 13:58:28.043948 9524 raft_consensus.cc:2889] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:28.041718 9978 consensus_queue.cc:237] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.048022 9491 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:28.014113 10430 raft_consensus.cc:3055] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:28.049096 10170 raft_consensus.cc:2749] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:28.049522 9594 raft_consensus.cc:2393] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:28.050554 9594 raft_consensus.cc:1240] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 6 LEADER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:27.999349 10319 consensus_queue.cc:1059] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260503 13:58:28.030167 10295 raft_consensus.cc:2804] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:28.055401 9418 leader_election.cc:304] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:28.056723 9566 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:28.058260 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:27.981964 10432 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.033452 10260 raft_consensus.cc:2749] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:28.040279 10419 raft_consensus.cc:2804] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:28.004069 10425 raft_consensus.cc:515] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.062321 9416 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:28.049360 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:28.053519 10385 raft_consensus.cc:2804] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:28.053077 10430 raft_consensus.cc:740] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:27.989748 10441 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.057770 10017 raft_consensus.cc:2749] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260503 13:58:28.030117 10248 consensus_queue.cc:1059] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260503 13:58:28.065842 10318 raft_consensus.cc:2804] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:28.069105 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:28.069653 10455 log.cc:927] Time spent T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.116s user 0.002s sys 0.000s
I20260503 13:58:28.070860 10253 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:28.071918 10193 raft_consensus.cc:515] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.055136 10420 consensus_queue.cc:1048] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260503 13:58:28.030738 10310 consensus_queue.cc:237] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.059759 10184 consensus_queue.cc:1048] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
W20260503 13:58:28.072727 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 56606.6us
I20260503 13:58:28.033219 10362 consensus_queue.cc:1059] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260503 13:58:28.057423 10295 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:28.079866 10318 raft_consensus.cc:493] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:28.072752 10453 raft_consensus.cc:2749] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:28.062875 10445 consensus_queue.cc:1059] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.021s
I20260503 13:58:28.064581 10243 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:28.082718 10319 raft_consensus.cc:3055] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:28.080159 10419 raft_consensus.cc:697] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
W20260503 13:58:28.095515 10454 log.cc:927] Time spent T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20260503 13:58:28.095705 10419 consensus_queue.cc:237] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:28.093271 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:28.066107 10238 consensus_queue.cc:1048] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260503 13:58:28.084103 10253 raft_consensus.cc:2749] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:28.097939 10319 raft_consensus.cc:740] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:28.081624 10260 consensus_queue.cc:1059] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260503 13:58:28.099309 9599 raft_consensus.cc:2393] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:28.094619 9567 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:28.103000 9416 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:28.081202 10304 consensus_queue.cc:1048] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.017s
I20260503 13:58:28.091514 10295 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:28.093570 10318 raft_consensus.cc:3060] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:28.098961 10362 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260503 13:58:28.122959 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:28.103718 10445 consensus_queue.cc:1243] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer 4bea375a55034750a1f8804b0db97868 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20260503 13:58:28.129642 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 72988.6us
I20260503 13:58:28.131242 10396 leader_election.cc:290] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:28.133801 9566 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:28.135957 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:28.131788 10447 log.cc:927] Time spent T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.133s user 0.000s sys 0.001s
I20260503 13:58:28.107924 9596 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:28.128703 10362 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:28.091599 10347 consensus_queue.cc:237] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.141561 10459 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:28.141927 10459 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:28.142235 10459 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:28.143028 10435 raft_consensus.cc:2749] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:28.092793 10113 consensus_queue.cc:237] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.141010 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:28.145413 9567 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:28.146220 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:28.146744 9451 raft_consensus.cc:2468] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:28.143967 9644 tablet_bootstrap.cc:492] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:28.147485 9644 ts_tablet_manager.cc:1403] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.516s user 0.016s sys 0.010s
I20260503 13:58:28.100916 10319 consensus_queue.cc:260] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.149732 9642 tablet_bootstrap.cc:492] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:28.150161 10319 raft_consensus.cc:3060] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
W20260503 13:58:28.144721 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 82005.8us
I20260503 13:58:28.149255 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:28.137518 10445 raft_consensus.cc:3055] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:28.136729 10396 raft_consensus.cc:2749] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:28.153293 9449 raft_consensus.cc:1275] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:28.153748 9567 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:28.154335 9978 consensus_queue.cc:1048] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:58:28.136823 10193 leader_election.cc:290] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:28.098727 10430 consensus_queue.cc:260] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:28.162036 9418 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:28.161077 9445 raft_consensus.cc:3055] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:28.143786 10362 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:28.165520 9450 raft_consensus.cc:1275] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:28.165584 9445 raft_consensus.cc:740] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:28.143563 10420 raft_consensus.cc:2749] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:28.178238 9567 leader_election.cc:304] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:28.150852 10304 raft_consensus.cc:493] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:28.187116 9978 raft_consensus.cc:2749] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:28.187685 9978 raft_consensus.cc:2749] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:28.167676 9445 consensus_queue.cc:260] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:28.191969 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:28.138366 10453 consensus_queue.cc:1048] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260503 13:58:28.156145 10445 raft_consensus.cc:740] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:28.171021 9520 raft_consensus.cc:1275] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:58:28.171680 10425 leader_election.cc:290] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:28.173099 10441 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:28.217617 10163 consensus_queue.cc:1059] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 0, Time since last communication: 0.007s
I20260503 13:58:28.225201 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:28.228891 10452 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:28.040580 9519 rpcz_store.cc:276] 0503 13:58:26.949185 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.558984 (+609799us) service_pool.cc:224] Handling call
0503 13:58:27.587836 (+ 28852us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:27.619616 (+ 31780us) raft_consensus.cc:1534] Early marking committed up to index 1
0503 13:58:27.704896 (+ 85280us) log.cc:844] Serialized 14 byte log entry
0503 13:58:27.972235 (+267339us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:27.975052 (+ 2817us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:27.979537 (+ 4485us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.982488 (+ 2951us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:27.999100 (+ 16612us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18024,"prepare.queue_time_us":65059,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":43912,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":16196,"thread_start_us":221764,"threads_started":2,"wal-append.queue_time_us":159680}
I20260503 13:58:28.229823 10445 consensus_queue.cc:260] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.189829 10304 raft_consensus.cc:515] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.188623 10193 consensus_queue.cc:1048] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260503 13:58:28.188608 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:28.161369 10427 consensus_queue.cc:1059] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
W20260503 13:58:28.246785 9418 leader_election.cc:343] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:28.237063 9445 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:28.238816 10457 raft_consensus.cc:2749] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:28.213634 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:28.240664 10458 log.cc:927] Time spent T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
I20260503 13:58:28.240578 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:28.173588 10340 consensus_queue.cc:1048] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20260503 13:58:28.239475 10452 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.243887 10445 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:28.244822 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:28.254839 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 151912us
I20260503 13:58:28.241528 10430 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:28.257160 9416 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:28.258989 9642 tablet_bootstrap.cc:654] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:28.239799 10432 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:28.222488 10017 consensus_queue.cc:1048] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
W20260503 13:58:28.254944 10428 consensus_meta.cc:287] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.682s user 0.004s sys 0.005s
I20260503 13:58:28.249869 9446 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:28.275209 9491 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
W20260503 13:58:28.278851 9491 leader_election.cc:343] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:28.279491 9491 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:28.282285 9491 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:28.289355 10251 raft_consensus.cc:2804] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:28.291064 10251 raft_consensus.cc:697] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:28.291376 10461 consensus_queue.cc:1048] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260503 13:58:28.291870 10251 consensus_queue.cc:237] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.284739 9644 raft_consensus.cc:359] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.293098 9644 raft_consensus.cc:385] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:28.293160 9416 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:28.293427 9644 raft_consensus.cc:740] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:28.294190 9416 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:28.248549 10238 consensus_queue.cc:1048] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260503 13:58:28.260710 9596 raft_consensus.cc:1240] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:28.256793 10385 raft_consensus.cc:697] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:28.293974 9644 consensus_queue.cc:260] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.285420 10451 raft_consensus.cc:3055] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:28.309065 10451 raft_consensus.cc:740] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
W20260503 13:58:28.309424 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 92748.1us
I20260503 13:58:28.269134 10361 raft_consensus.cc:2749] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:28.273415 10420 raft_consensus.cc:2749] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:28.303681 9445 raft_consensus.cc:1275] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260503 13:58:28.310247 9566 leader_election.cc:304] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:28.295486 9416 leader_election.cc:304] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:28.314905 10385 consensus_queue.cc:237] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.288698 10163 raft_consensus.cc:2749] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:28.316615 9525 raft_consensus.cc:1240] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:28.259085 10434 consensus_queue.cc:1059] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260503 13:58:28.324970 10340 raft_consensus.cc:2804] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:28.325460 10340 raft_consensus.cc:697] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:28.309847 10310 raft_consensus.cc:2749] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:28.327061 9416 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:28.311462 10461 raft_consensus.cc:2804] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election won for term 4
W20260503 13:58:28.330649 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:28.333873 9445 raft_consensus.cc:1275] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:28.335256 9644 ts_tablet_manager.cc:1434] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.188s user 0.006s sys 0.000s
I20260503 13:58:28.335883 10434 raft_consensus.cc:3055] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:28.336232 9644 tablet_bootstrap.cc:492] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:28.336238 10434 raft_consensus.cc:740] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:28.337488 10434 consensus_queue.cc:260] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.338568 10434 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:28.326653 10199 raft_consensus.cc:3060] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:28.290246 10362 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.319437 10474 raft_consensus.cc:2804] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:28.326234 10340 consensus_queue.cc:237] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.348829 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:28.349424 9523 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 372) took 1947 ms (client timeout 2372 ms). Trace:
I20260503 13:58:28.329409 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:28.353969 9643 tablet_bootstrap.cc:492] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:28.361372 9643 ts_tablet_manager.cc:1403] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.461s user 0.018s sys 0.018s
I20260503 13:58:28.360818 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:28.355239 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:28.356118 10304 leader_election.cc:290] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:28.356431 10430 consensus_queue.cc:1048] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:28.344332 10193 consensus_queue.cc:1048] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.013s
I20260503 13:58:28.357285 9452 raft_consensus.cc:1275] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260503 13:58:28.357892 10452 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:28.359288 9523 rpcz_store.cc:269] 0503 13:58:26.402327 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:26.718811 (+316484us) service_pool.cc:224] Handling call
0503 13:58:28.349219 (+1630408us) spinlock_profiling.cc:243] Waited 685 ms on lock 0x7b54003d40d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:28.349349 (+ 130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1575308672}
I20260503 13:58:28.346036 10420 raft_consensus.cc:2749] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:28.363868 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:28.350255 10474 raft_consensus.cc:493] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:28.363447 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:28.365320 9519 raft_consensus.cc:3060] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:28.376868 9644 tablet_bootstrap.cc:654] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:28.349385 10459 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.369434 10193 consensus_queue.cc:1243] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer 1cb63886040047559d3893923cc9aaf6 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260503 13:58:28.372355 10424 raft_consensus.cc:3055] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:28.359297 9446 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:28.376497 9526 raft_consensus.cc:2393] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:28.379540 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:28.376655 10474 raft_consensus.cc:3060] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:28.364084 10361 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:28.362068 10461 raft_consensus.cc:697] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:28.412073 9567 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:28.370352 9527 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:28.415504 9643 raft_consensus.cc:359] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.415836 9566 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:28.416998 9451 raft_consensus.cc:1275] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260503 13:58:28.417675 9643 raft_consensus.cc:385] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:28.418170 9643 raft_consensus.cc:740] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:28.381989 10424 raft_consensus.cc:740] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:28.420292 9416 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:28.436419 9418 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:28.413106 10362 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:28.432001 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:28.419176 9643 consensus_queue.cc:260] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.411561 10361 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.436612 10170 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:28.436266 10193 raft_consensus.cc:2804] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:28.451638 10470 consensus_queue.cc:1048] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.043s
I20260503 13:58:28.421569 10420 consensus_queue.cc:1048] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260503 13:58:28.455930 10424 consensus_queue.cc:260] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.421310 10347 consensus_queue.cc:1048] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260503 13:58:28.465296 9446 raft_consensus.cc:1217] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[2.2-5.4] Dedup: 2.2->[3.3-5.4]
W20260503 13:58:28.477092 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:28.442484 10419 raft_consensus.cc:2804] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:28.479627 9643 ts_tablet_manager.cc:1434] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.118s user 0.007s sys 0.002s
I20260503 13:58:28.477874 9446 pending_rounds.cc:85] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6: Aborting all ops after (but not including) 2
I20260503 13:58:28.451943 9527 raft_consensus.cc:2468] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:28.484947 9446 pending_rounds.cc:107] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260503 13:58:28.486553 9446 raft_consensus.cc:2889] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:28.479205 10419 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:28.478334 10170 raft_consensus.cc:697] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:28.511332 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:28.479014 10193 raft_consensus.cc:697] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:28.514911 10361 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:28.516572 10465 log.cc:927] Time spent T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.138s user 0.000s sys 0.001s
I20260503 13:58:28.516839 10459 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:28.521138 9449 raft_consensus.cc:2393] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:28.526886 9519 raft_consensus.cc:2468] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:28.512737 10475 raft_consensus.cc:2749] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:28.513235 10424 raft_consensus.cc:3060] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:28.513055 10419 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:28.483938 10295 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.553500 9519 raft_consensus.cc:1275] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260503 13:58:28.476950 10461 consensus_queue.cc:237] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.565743 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:28.568101 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:28.568282 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:28.570429 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 342) took 1066 ms. Trace:
I20260503 13:58:28.570775 9453 rpcz_store.cc:276] 0503 13:58:27.504245 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.554858 (+1050613us) service_pool.cc:224] Handling call
0503 13:58:28.570373 (+ 15515us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:28.575230 10476 log.cc:927] Time spent T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20260503 13:58:28.576154 9642 tablet_bootstrap.cc:492] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:28.576609 9642 ts_tablet_manager.cc:1403] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.427s user 0.016s sys 0.006s
I20260503 13:58:28.559983 10361 raft_consensus.cc:493] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:28.545608 10318 raft_consensus.cc:515] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.580205 9595 raft_consensus.cc:2393] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:28.580746 9527 raft_consensus.cc:1275] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260503 13:58:28.581521 10361 raft_consensus.cc:515] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.587846 9445 raft_consensus.cc:1240] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260503 13:58:28.589810 9453 raft_consensus.cc:1275] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260503 13:58:28.590143 9453 pending_rounds.cc:85] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6: Aborting all ops after (but not including) 2
I20260503 13:58:28.590147 9642 raft_consensus.cc:359] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.590626 9453 pending_rounds.cc:107] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260503 13:58:28.590840 9642 raft_consensus.cc:385] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:28.590876 9453 raft_consensus.cc:2889] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:28.591254 9642 raft_consensus.cc:740] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:28.592162 9642 consensus_queue.cc:260] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.595212 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:28.565697 10470 consensus_queue.cc:1048] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260503 13:58:28.578327 10193 consensus_queue.cc:237] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.596648 10475 consensus_queue.cc:1048] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260503 13:58:28.591697 10468 consensus_queue.cc:1048] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260503 13:58:28.617623 9598 raft_consensus.cc:1275] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:28.618925 9642 ts_tablet_manager.cc:1434] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20260503 13:58:28.614285 9448 raft_consensus.cc:2393] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:28.579128 10170 consensus_queue.cc:237] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.614903 10361 leader_election.cc:290] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:28.636104 9600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 282) took 1250 ms. Trace:
I20260503 13:58:28.637616 9600 rpcz_store.cc:276] 0503 13:58:27.385785 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.393829 (+ 8044us) service_pool.cc:224] Handling call
0503 13:58:28.635944 (+1242115us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b54002ba7d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:28.636050 (+ 106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":76441472}
W20260503 13:58:28.637980 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:28.653960 9642 tablet_bootstrap.cc:492] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:28.617097 9593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 289) took 1227 ms. Trace:
I20260503 13:58:28.660842 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:28.691792 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 282524us
I20260503 13:58:28.692756 9567 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:28.695451 9600 raft_consensus.cc:2410] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260503 13:58:28.694222 9567 leader_election.cc:343] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:28.694545 9602 raft_consensus.cc:1275] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:28.665900 9593 rpcz_store.cc:276] 0503 13:58:27.390052 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.394616 (+ 4564us) service_pool.cc:224] Handling call
0503 13:58:28.616918 (+1222302us) spinlock_profiling.cc:243] Waited 85.6 ms on lock 0x7b54002bacd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:28.617040 (+ 122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":196949376}
I20260503 13:58:28.694805 10457 raft_consensus.cc:2749] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:28.700058 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:28.701061 9567 leader_election.cc:304] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:28.701540 9593 raft_consensus.cc:2468] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:28.703528 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:28.704485 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:28.705013 9593 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:28.701735 10420 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:28.705802 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:28.696530 10304 raft_consensus.cc:493] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:28.706681 10304 raft_consensus.cc:515] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:28.706265 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 67351.9us
I20260503 13:58:28.709381 9491 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
W20260503 13:58:28.710146 9416 leader_election.cc:343] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:28.709431 9567 leader_election.cc:304] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:28.710556 9416 leader_election.cc:304] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:28.710245 9491 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:28.711894 9567 leader_election.cc:304] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:28.631603 10474 raft_consensus.cc:515] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.690933 10295 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:28.714447 10163 raft_consensus.cc:2804] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260503 13:58:28.699518 10477 log.cc:927] Time spent T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260503 13:58:28.722972 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:28.725755 9520 raft_consensus.cc:2393] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:28.708442 10304 leader_election.cc:290] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:28.713310 10457 raft_consensus.cc:2749] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:28.717192 10434 raft_consensus.cc:2804] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:28.718667 10475 raft_consensus.cc:2804] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:28.720679 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:28.720219 10467 raft_consensus.cc:2749] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:28.722220 10113 raft_consensus.cc:2804] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:28.724370 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:28.722335 10163 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
W20260503 13:58:28.733553 10372 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260503 13:58:28.729453 10362 raft_consensus.cc:493] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:28.730525 10318 leader_election.cc:290] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:28.750541 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:28.758189 9520 raft_consensus.cc:2468] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:28.705827 10420 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.759114 9567 leader_election.cc:304] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:28.734356 10434 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:28.735330 10475 raft_consensus.cc:493] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:28.761861 9492 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:28.762866 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:28.765204 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:28.761757 10434 raft_consensus.cc:3060] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
W20260503 13:58:28.765957 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:28.762899 9492 leader_election.cc:343] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:28.768941 9492 leader_election.cc:304] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:28.756788 10163 raft_consensus.cc:3060] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:28.767186 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:28.771515 9520 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:28.757748 9593 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:28.761462 10475 raft_consensus.cc:3060] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:28.759517 10251 consensus_queue.cc:1059] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260503 13:58:28.764019 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:28.777745 9600 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:28.765637 10432 raft_consensus.cc:493] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:28.770509 10358 raft_consensus.cc:3055] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:28.766017 10428 raft_consensus.cc:2749] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:28.759653 10453 consensus_queue.cc:1048] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
W20260503 13:58:28.786749 9566 leader_election.cc:343] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:28.786167 10432 raft_consensus.cc:515] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.771886 10457 raft_consensus.cc:2804] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:28.787302 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:28.766286 10113 raft_consensus.cc:697] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:28.760732 10362 raft_consensus.cc:515] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.795619 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:28.797179 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:28.797902 9447 raft_consensus.cc:2393] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 5: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:28.798511 9642 tablet_bootstrap.cc:654] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:28.798600 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:28.799690 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:28.801914 10432 leader_election.cc:290] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:28.805137 10483 log.cc:927] Time spent T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260503 13:58:28.806175 9520 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:28.787734 10474 leader_election.cc:290] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:28.791553 10457 raft_consensus.cc:493] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:28.807039 9567 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:28.791235 10251 raft_consensus.cc:2749] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:28.788653 10358 raft_consensus.cc:740] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:28.811726 10457 raft_consensus.cc:3060] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:28.818049 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:28.818545 10283 raft_consensus.cc:3055] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:28.818900 10283 raft_consensus.cc:740] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:28.819202 10362 leader_election.cc:290] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:28.819959 9522 raft_consensus.cc:1275] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:28.819680 10283 consensus_queue.cc:260] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.807435 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:28.808995 9444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 320) took 3347 ms. Trace:
I20260503 13:58:28.809368 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:28.831651 10362 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:28.937144 10420 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:28.941205 9601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 288) took 1551 ms. Trace:
I20260503 13:58:28.941655 9601 rpcz_store.cc:276] 0503 13:58:27.389772 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.394857 (+ 5085us) service_pool.cc:224] Handling call
0503 13:58:28.934457 (+1539600us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b5400078d54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:28.941140 (+ 6683us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":309823744}
I20260503 13:58:28.943715 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:28.944295 9600 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:28.826224 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:28.818044 10113 consensus_queue.cc:237] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.946537 9601 raft_consensus.cc:1275] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:58:28.951684 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:28.956429 9520 raft_consensus.cc:2468] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:28.952749 9528 raft_consensus.cc:1240] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:28.957422 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 391) took 1747 ms. Trace:
I20260503 13:58:28.957715 9528 rpcz_store.cc:276] 0503 13:58:27.210016 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.639434 (+429418us) service_pool.cc:224] Handling call
0503 13:58:28.952370 (+1312936us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b5800004598. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:28.952545 (+ 175us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:28.957212 (+ 4667us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:28.957365 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7125248}
I20260503 13:58:28.834116 10362 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.941606 10467 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:28.933712 9522 raft_consensus.cc:1275] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260503 13:58:28.976624 9598 raft_consensus.cc:1275] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260503 13:58:28.977000 9598 pending_rounds.cc:85] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868: Aborting all ops after (but not including) 1
I20260503 13:58:28.977195 9598 pending_rounds.cc:107] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260503 13:58:28.977411 9598 raft_consensus.cc:2889] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260503 13:58:28.958366 10238 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.507s user 0.023s sys 0.048s
I20260503 13:58:28.947669 10304 raft_consensus.cc:2804] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:28.976657 9522 pending_rounds.cc:85] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a: Aborting all ops after (but not including) 1
I20260503 13:58:28.942647 10163 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.989285 9593 raft_consensus.cc:2393] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:28.978948 10420 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:28.997395 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 293) took 1573 ms. Trace:
I20260503 13:58:28.997949 9597 rpcz_store.cc:276] 0503 13:58:27.424485 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.456425 (+ 31940us) service_pool.cc:224] Handling call
0503 13:58:28.997197 (+1540772us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b54002b98d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:28.997340 (+ 143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1009780480}
I20260503 13:58:28.946111 10358 consensus_queue.cc:260] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.991393 9522 pending_rounds.cc:107] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260503 13:58:28.987066 10304 raft_consensus.cc:697] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:29.003265 9600 raft_consensus.cc:1240] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260503 13:58:28.980221 10467 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.988963 10475 raft_consensus.cc:515] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.014488 9522 raft_consensus.cc:2889] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:29.050690 10028 raft_consensus.cc:493] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:29.051237 10028 raft_consensus.cc:515] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.053174 10028 leader_election.cc:290] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:29.075380 10304 consensus_queue.cc:237] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.075961 9602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 302) took 1568 ms. Trace:
I20260503 13:58:29.076365 9602 rpcz_store.cc:276] 0503 13:58:27.508003 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.547577 (+ 39574us) service_pool.cc:224] Handling call
0503 13:58:28.653787 (+1106210us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:28.698907 (+ 45120us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:29.075704 (+376797us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54002ba7d0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:29.075876 (+ 172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2502631296}
I20260503 13:58:29.030340 10358 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:29.001286 10420 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:28.825733 9444 rpcz_store.cc:276] 0503 13:58:25.461644 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:27.064223 (+1602579us) service_pool.cc:224] Handling call
0503 13:58:27.138375 (+ 74152us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:27.138811 (+ 436us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:27.138844 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:27.139393 (+ 549us) log.cc:844] Serialized 26 byte log entry
0503 13:58:27.500712 (+361319us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:27.503972 (+ 3260us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:27.510707 (+ 6735us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:27.991573 (+480866us) spinlock_profiling.cc:243] Waited 7.16 ms on lock 0x7b48000f0558. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f46a631a1 00007f9f46a5a237 00007f9f46a5a0ca 00007f9f46b5192f 00007f9f46b515f7 00007f9f46b65c1c 00007f9f46b65bc8 00007f9f46b65b48 00007f9f46b65b03 00007f9f46b64dac 00007f9f45a8db36
0503 13:58:28.807924 (+816351us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b24000ef228. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e8dfe94 00007f9f3e8ddf9b 00007f9f3e96cd39 00007f9f3e972c75 00007f9f3e972b42 00007f9f3e972bce 00007f9f3e3a661b 00007f9f46ae30ba 00007f9f3e972adc 00007f9f3e96519d
0503 13:58:28.808771 (+ 847us) raft_consensus.cc:1685] finished
0503 13:58:28.808797 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:28.808922 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12386,"raft.run_cpu_time_us":9810,"raft.run_wall_time_us":834118,"spinlock_wait_cycles":64450048,"thread_start_us":332373,"threads_started":1,"wal-append.queue_time_us":447420}
W20260503 13:58:29.099960 10491 log.cc:927] Time spent T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260503 13:58:29.105245 10163 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:29.108851 9644 tablet_bootstrap.cc:492] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:29.112084 9644 ts_tablet_manager.cc:1403] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.776s user 0.013s sys 0.016s
I20260503 13:58:29.110316 10420 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:29.095125 10451 consensus_queue.cc:260] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:29.111275 10419 consensus_meta.cc:287] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.552s user 0.005s sys 0.004s
I20260503 13:58:29.124318 9491 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:29.126462 9491 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:29.127410 10358 raft_consensus.cc:2804] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:29.127784 10358 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:29.128085 10358 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:29.110338 10362 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:29.105619 10467 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:29.130679 9416 leader_election.cc:304] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:29.123596 10340 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.012s sys 0.024s
I20260503 13:58:29.125356 10451 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
W20260503 13:58:29.133669 10340 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.012s sys 0.024s
W20260503 13:58:29.143136 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:29.130728 10163 raft_consensus.cc:2804] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:29.140638 9416 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:29.140007 10028 consensus_queue.cc:1048] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260503 13:58:29.154606 10163 raft_consensus.cc:697] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:29.117118 10419 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.156872 10318 consensus_queue.cc:1048] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260503 13:58:29.147264 10467 raft_consensus.cc:2749] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260503 13:58:29.169020 9416 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:29.171553 10163 consensus_queue.cc:237] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:29.172391 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:29.173659 10470 raft_consensus.cc:2749] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:29.174582 10475 leader_election.cc:290] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:29.178110 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:29.179703 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:29.186458 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:29.187147 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:29.151396 10434 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.197129 10435 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:29.197578 10435 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.199249 10435 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:29.200021 10420 consensus_queue.cc:1048] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:29.200346 9519 raft_consensus.cc:1275] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:29.202862 10457 raft_consensus.cc:515] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.190658 9644 raft_consensus.cc:359] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.204020 9643 tablet_bootstrap.cc:492] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:29.204063 9644 raft_consensus.cc:385] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:29.204476 9644 raft_consensus.cc:740] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:29.183473 10028 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:29.206413 10028 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.207485 10419 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:29.208122 9447 raft_consensus.cc:2393] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 5: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:29.205408 10470 raft_consensus.cc:2749] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260503 13:58:29.157281 10362 consensus_queue.cc:1048] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260503 13:58:29.205147 9644 consensus_queue.cc:260] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.177774 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:29.200426 9527 raft_consensus.cc:1240] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:29.205135 10457 leader_election.cc:290] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:29.208379 10028 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:29.212728 10424 consensus_meta.cc:287] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.657s user 0.007s sys 0.004s
W20260503 13:58:29.230566 9452 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.020s sys 0.060s
W20260503 13:58:29.249102 9452 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s user 0.020s sys 0.060s
I20260503 13:58:29.247918 9444 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
W20260503 13:58:29.271296 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 508374us
I20260503 13:58:29.285511 10434 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:29.292618 9446 raft_consensus.cc:1217] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[2.2-2.2] Dedup: 2.2->[]
I20260503 13:58:29.293087 9644 ts_tablet_manager.cc:1434] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.175s user 0.009s sys 0.004s
I20260503 13:58:29.279304 10028 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:29.310227 10028 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:29.311445 10472 log.cc:927] Time spent T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.802s user 0.000s sys 0.003s
W20260503 13:58:29.314967 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:29.315388 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:29.321171 9521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 339) took 1167 ms. Trace:
I20260503 13:58:29.321372 9644 tablet_bootstrap.cc:492] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:29.321513 9521 rpcz_store.cc:276] 0503 13:58:28.154070 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.154262 (+ 192us) service_pool.cc:224] Handling call
0503 13:58:28.173443 (+ 19181us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:28.173775 (+ 332us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:28.173800 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:28.174202 (+ 402us) log.cc:844] Serialized 26 byte log entry
0503 13:58:28.369833 (+195631us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:28.369868 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:28.369922 (+ 54us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:29.301385 (+931463us) spinlock_profiling.cc:243] Waited 6.04 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:29.312377 (+ 10992us) raft_consensus.cc:1685] finished
0503 13:58:29.312400 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:29.321112 (+ 8712us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1430,"raft.queue_time_us":14829,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":30993,"spinlock_wait_cycles":13883264,"thread_start_us":195320,"threads_started":1,"wal-append.queue_time_us":266133}
I20260503 13:58:29.312238 10028 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:29.319978 9446 raft_consensus.cc:1240] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260503 13:58:29.320143 10434 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:29.323508 10251 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:29.353390 10251 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.355340 10251 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:29.338155 10028 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:29.356994 10028 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.357658 10358 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.358749 10028 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:29.320317 10336 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.749s user 0.019s sys 0.028s
W20260503 13:58:29.326536 10493 log.cc:927] Time spent T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.301s user 0.001s sys 0.001s
I20260503 13:58:29.340046 9492 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:29.379981 9567 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.003s sys 0.016s
W20260503 13:58:29.383780 9567 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.573s user 0.003s sys 0.016s
W20260503 13:58:29.384114 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 577097us
W20260503 13:58:29.360384 10336 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.800s user 0.019s sys 0.028s
I20260503 13:58:29.353434 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:29.379873 10028 raft_consensus.cc:2749] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260503 13:58:29.415258 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:29.360149 10427 consensus_queue.cc:1059] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
W20260503 13:58:29.422252 9567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:29.415627 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 393) took 1277 ms. Trace:
I20260503 13:58:29.439613 9446 raft_consensus.cc:2468] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:29.360318 10434 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.386960 10358 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:29.458401 9593 raft_consensus.cc:1240] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20260503 13:58:29.416193 9444 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:29.470709 9492 leader_election.cc:304] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:29.443306 9567 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:29.472472 10336 raft_consensus.cc:3055] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Stepping down as leader of term 4
W20260503 13:58:29.481549 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:29.475725 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:29.487361 9567 leader_election.cc:343] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:29.513608 9528 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.023s sys 0.037s
W20260503 13:58:29.512670 10300 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.533s user 0.007s sys 0.016s
I20260503 13:58:29.516484 9978 consensus_queue.cc:1048] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:29.517786 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:29.514395 9528 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s user 0.023s sys 0.038s
I20260503 13:58:29.521060 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:29.480201 10170 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s user 0.010s sys 0.014s
W20260503 13:58:29.481230 10193 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.000s sys 0.024s
I20260503 13:58:29.475451 9596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 329) took 1505 ms. Trace:
W20260503 13:58:29.524677 9601 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.003s sys 0.019s
I20260503 13:58:29.513839 10358 raft_consensus.cc:2804] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260503 13:58:29.552881 9601 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s user 0.005s sys 0.020s
I20260503 13:58:29.513356 10396 raft_consensus.cc:2749] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260503 13:58:29.549894 9520 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.019s sys 0.098s
W20260503 13:58:29.553544 9520 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.555s user 0.019s sys 0.098s
I20260503 13:58:29.552302 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:29.554438 9521 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:29.484521 10336 raft_consensus.cc:740] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:29.441044 9448 rpcz_store.cc:276] 0503 13:58:28.138120 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.145423 (+ 7303us) service_pool.cc:224] Handling call
0503 13:58:29.415427 (+1270004us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b5400252e54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:29.415558 (+ 131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":789974784}
W20260503 13:58:29.556299 9566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.010s sys 0.021s
I20260503 13:58:29.549332 9519 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
W20260503 13:58:29.560799 9566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.750s user 0.010s sys 0.021s
I20260503 13:58:29.564090 9418 leader_election.cc:304] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:29.549762 10170 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s user 0.010s sys 0.014s
I20260503 13:58:29.566691 9519 raft_consensus.cc:1275] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260503 13:58:29.551414 10193 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s user 0.000s sys 0.024s
I20260503 13:58:29.517768 10166 consensus_queue.cc:1048] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20260503 13:58:29.552680 9596 rpcz_store.cc:276] 0503 13:58:27.964154 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.164430 (+200276us) service_pool.cc:224] Handling call
0503 13:58:28.211999 (+ 47569us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:28.311005 (+ 99006us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:29.442605 (+1131600us) spinlock_profiling.cc:243] Waited 38 ms on lock 0x7b54002594d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:29.469986 (+ 27381us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":87305984}
I20260503 13:58:29.587584 9596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 333) took 1002 ms. Trace:
I20260503 13:58:29.587937 9596 rpcz_store.cc:276] 0503 13:58:28.584846 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:29.587064 (+1002218us) service_pool.cc:224] Handling call
0503 13:58:29.587517 (+ 453us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:29.552292 10248 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:29.589296 10248 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.591203 10248 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:29.547608 10319 consensus_queue.cc:1059] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260503 13:58:29.548275 10184 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:29.593292 9596 raft_consensus.cc:1275] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:29.593700 10184 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.595139 10310 raft_consensus.cc:2804] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:29.595525 10310 raft_consensus.cc:493] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:29.595853 10310 raft_consensus.cc:3060] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:29.557368 10358 raft_consensus.cc:493] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:29.600850 10358 raft_consensus.cc:3060] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:29.548059 10300 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:29.603296 10300 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.605252 10300 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:29.606410 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:29.607012 9520 raft_consensus.cc:2468] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:29.610503 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:29.612852 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:29.613437 9449 raft_consensus.cc:2468] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
W20260503 13:58:29.635522 9598 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.011s sys 0.015s
W20260503 13:58:29.636478 9598 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s user 0.011s sys 0.015s
I20260503 13:58:29.640573 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 400) took 1307 ms. Trace:
I20260503 13:58:29.640941 9451 rpcz_store.cc:276] 0503 13:58:28.332681 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.341312 (+ 8631us) service_pool.cc:224] Handling call
0503 13:58:28.341589 (+ 277us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:28.417336 (+ 75747us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:29.640327 (+1222991us) spinlock_profiling.cc:243] Waited 72.7 ms on lock 0x7b5400303254. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:29.640499 (+ 172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":167154560}
I20260503 13:58:29.642019 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:29.643221 10498 log.cc:927] Time spent T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260503 13:58:29.644392 9524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 350) took 1293 ms. Trace:
I20260503 13:58:29.644747 9524 rpcz_store.cc:276] 0503 13:58:28.350585 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.597453 (+246868us) service_pool.cc:224] Handling call
0503 13:58:28.610711 (+ 13258us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:28.610999 (+ 288us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:28.611027 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:28.632380 (+ 21353us) log.cc:844] Serialized 26 byte log entry
0503 13:58:29.171802 (+539422us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:29.171839 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:29.171899 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:29.644200 (+472301us) raft_consensus.cc:1685] finished
0503 13:58:29.644220 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:29.644338 (+ 118us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":70,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":2646,"thread_start_us":472168,"threads_started":1,"wal-append.queue_time_us":784195}
I20260503 13:58:29.646312 9522 raft_consensus.cc:1240] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:29.647063 9594 raft_consensus.cc:1240] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:29.524291 10463 consensus_queue.cc:1048] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
I20260503 13:58:29.559465 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:29.562742 10445 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:29.607955 10396 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:29.610072 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:29.647621 9602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.020s sys 0.023s
I20260503 13:58:29.629766 10170 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:29.705579 10287 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.021s sys 0.063s
W20260503 13:58:29.705703 9568 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.019s sys 0.049s
I20260503 13:58:29.684006 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:29.706087 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 337) took 1552 ms. Trace:
W20260503 13:58:29.836733 9526 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.017s sys 0.029s
I20260503 13:58:29.990996 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:29.761148 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:29.626724 10028 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:29.988869 9453 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.753s user 0.007s sys 0.012s
W20260503 13:58:29.988718 9642 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.871s user 0.012s sys 0.017s
W20260503 13:58:29.730746 9597 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.709s user 0.035s sys 0.030s
I20260503 13:58:29.920800 9524 raft_consensus.cc:1275] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260503 13:58:29.994938 9453 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.759s user 0.007s sys 0.012s
I20260503 13:58:29.995231 9525 rpcz_store.cc:276] 0503 13:58:28.153518 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.160669 (+ 7151us) service_pool.cc:224] Handling call
0503 13:58:28.171387 (+ 10718us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:28.316973 (+145586us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:29.705857 (+1388884us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54001559d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:29.706032 (+ 175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":333506816}
W20260503 13:58:29.990551 9568 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.854s user 0.036s sys 0.054s
W20260503 13:58:29.993237 9597 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.972s user 0.037s sys 0.048s
W20260503 13:58:29.992056 9526 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.789s user 0.019s sys 0.039s
I20260503 13:58:29.957628 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:29.992741 9642 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.875s user 0.013s sys 0.017s
W20260503 13:58:29.689085 9565 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.009s sys 0.052s
W20260503 13:58:29.848587 10462 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.009s sys 0.017s
I20260503 13:58:29.990942 9449 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
W20260503 13:58:29.681435 10113 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.000s sys 0.062s
W20260503 13:58:29.689837 10473 log.cc:927] Time spent T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
W20260503 13:58:29.959811 9602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.882s user 0.022s sys 0.038s
I20260503 13:58:29.676998 10300 consensus_queue.cc:1048] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.334s
I20260503 13:58:29.885673 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:30.560397 9525 raft_consensus.cc:1275] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260503 13:58:30.142038 10199 raft_consensus.cc:493] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:30.635643 10199 raft_consensus.cc:515] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.322685 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 354) took 1629 ms. Trace:
I20260503 13:58:30.637689 9528 rpcz_store.cc:276] 0503 13:58:28.693299 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.958552 (+265253us) service_pool.cc:224] Handling call
0503 13:58:28.980696 (+ 22144us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:28.983321 (+ 2625us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:28.983351 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:28.983781 (+ 430us) log.cc:844] Serialized 26 byte log entry
0503 13:58:29.519606 (+535825us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:29.520263 (+ 657us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:29.520329 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:29.648737 (+128408us) spinlock_profiling.cc:243] Waited 2.97 ms on lock 0x7b48000fc0d8. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f46a631a1 00007f9f46a5a237 00007f9f46a5a0ca 00007f9f46b5192f 00007f9f46b515f7 00007f9f46b65c1c 00007f9f46b65bc8 00007f9f46b65b48 00007f9f46b65b03 00007f9f46b64dac 00007f9f45a8db36
0503 13:58:30.322402 (+673665us) raft_consensus.cc:1685] finished
0503 13:58:30.322424 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:30.322619 (+ 195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2262,"raft.queue_time_us":2941,"raft.run_cpu_time_us":1876,"raft.run_wall_time_us":690968,"spinlock_wait_cycles":6829952,"thread_start_us":514862,"threads_started":1,"wal-append.queue_time_us":1036155}
I20260503 13:58:29.956745 10434 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:30.639103 10434 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.140396 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:30.641196 9519 raft_consensus.cc:3060] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:30.640547 9525 raft_consensus.cc:3055] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:30.641810 9525 raft_consensus.cc:740] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:30.642607 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:30.103170 9449 raft_consensus.cc:1275] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260503 13:58:30.607869 9492 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.004s sys 0.004s
W20260503 13:58:30.643329 9492 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s user 0.004s sys 0.004s
I20260503 13:58:30.168823 10511 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:30.644994 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:30.645212 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:30.645645 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:30.646219 9449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 412) took 1916 ms. Trace:
I20260503 13:58:30.646648 9449 rpcz_store.cc:276] 0503 13:58:28.729790 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.012290 (+1282500us) service_pool.cc:224] Handling call
0503 13:58:30.087678 (+ 75388us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:30.645928 (+558250us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:30.646160 (+ 232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:29.682464 9527 raft_consensus.cc:1275] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:30.364868 9444 raft_consensus.cc:1240] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: []
W20260503 13:58:30.319187 10163 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.860s user 0.025s sys 0.028s
I20260503 13:58:29.993384 10396 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.806649 10361 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.642472 9525 consensus_queue.cc:260] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:30.607534 10310 consensus_meta.cc:287] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 1.011s user 0.008s sys 0.001s
I20260503 13:58:30.068611 9451 raft_consensus.cc:1275] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260503 13:58:30.388075 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 328) took 1845 ms. Trace:
I20260503 13:58:30.660802 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 353) took 1959 ms. Trace:
I20260503 13:58:30.010943 10170 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.011523 10028 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:29.897827 10503 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
W20260503 13:58:30.033006 10462 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.853s user 0.020s sys 0.017s
I20260503 13:58:29.708573 10445 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:30.561303 10260 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.003s sys 0.026s
W20260503 13:58:30.011487 9565 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.880s user 0.017s sys 0.082s
I20260503 13:58:30.341574 9601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 325) took 1853 ms. Trace:
I20260503 13:58:30.652776 10540 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.685694 9601 rpcz_store.cc:276] 0503 13:58:28.487756 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.959813 (+472057us) service_pool.cc:224] Handling call
0503 13:58:28.979628 (+ 19815us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:28.985856 (+ 6228us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:28.985884 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:28.993922 (+ 8038us) log.cc:844] Serialized 26 byte log entry
0503 13:58:29.553270 (+559348us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:29.553309 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:29.553373 (+ 64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:30.341358 (+787985us) raft_consensus.cc:1685] finished
0503 13:58:30.341380 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:30.341514 (+ 134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":210017,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":13976,"thread_start_us":550397,"threads_started":1,"wal-append.queue_time_us":1067366}
W20260503 13:58:30.565662 9491 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.004s sys 0.009s
W20260503 13:58:30.687608 9491 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.677s user 0.005s sys 0.009s
W20260503 13:58:30.631958 10113 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.521s user 0.015s sys 0.093s
I20260503 13:58:29.690402 10283 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:30.641263 9528 raft_consensus.cc:1275] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260503 13:58:30.655669 9643 tablet_bootstrap.cc:654] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:30.689971 9528 pending_rounds.cc:85] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a: Aborting all ops after (but not including) 1
W20260503 13:58:30.690019 9491 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:30.690410 9528 pending_rounds.cc:107] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260503 13:58:30.690753 9528 raft_consensus.cc:2889] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:30.645268 10511 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.660473 10535 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:30.005640 10287 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.983s user 0.031s sys 0.078s
I20260503 13:58:30.661156 9451 rpcz_store.cc:276] 0503 13:58:28.701775 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.010804 (+1309029us) service_pool.cc:224] Handling call
0503 13:58:30.068262 (+ 57458us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:30.660552 (+592290us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:30.660754 (+ 202us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:30.684872 10260 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.772s user 0.004s sys 0.027s
W20260503 13:58:30.331522 10251 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.862s user 0.048s sys 0.073s
I20260503 13:58:30.663705 9525 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
W20260503 13:58:30.319020 10427 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.847s user 0.016s sys 0.038s
W20260503 13:58:30.658272 10163 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.198s user 0.029s sys 0.031s
I20260503 13:58:30.650645 9519 raft_consensus.cc:2468] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:30.683347 9444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 413) took 1912 ms. Trace:
I20260503 13:58:30.683334 9642 tablet_bootstrap.cc:492] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:30.089035 10510 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
W20260503 13:58:30.630208 10419 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.007s sys 0.006s
W20260503 13:58:30.691241 9491 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:30.640426 10434 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.692137 9521 raft_consensus.cc:1275] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260503 13:58:30.698069 10251 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.228s user 0.069s sys 0.137s
W20260503 13:58:30.700196 10427 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.227s user 0.016s sys 0.045s
I20260503 13:58:30.656836 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:30.659988 9448 raft_consensus.cc:2468] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:29.690302 10336 consensus_queue.cc:260] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.660717 9598 rpcz_store.cc:276] 0503 13:58:28.542980 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.978159 (+435179us) service_pool.cc:224] Handling call
0503 13:58:28.987500 (+ 9341us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:28.987701 (+ 201us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:28.987725 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:28.988105 (+ 380us) log.cc:844] Serialized 26 byte log entry
0503 13:58:29.637446 (+649341us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:29.637486 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:29.637552 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:30.387866 (+750314us) raft_consensus.cc:1685] finished
0503 13:58:30.387887 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:30.388020 (+ 133us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":615346,"threads_started":1,"wal-append.queue_time_us":1346118}
I20260503 13:58:30.696537 9451 raft_consensus.cc:1240] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260503 13:58:30.706048 9524 raft_consensus.cc:2468] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:30.695181 9446 raft_consensus.cc:2410] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260503 13:58:30.683327 9527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 361) took 1200 ms. Trace:
I20260503 13:58:30.699429 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:30.745576 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 358) took 1472 ms. Trace:
I20260503 13:58:30.745874 9451 rpcz_store.cc:276] 0503 13:58:29.273166 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.695621 (+1422455us) service_pool.cc:224] Handling call
0503 13:58:30.696340 (+ 719us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:30.745326 (+ 48986us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:30.745530 (+ 204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:30.746259 9519 raft_consensus.cc:1275] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260503 13:58:30.664635 10361 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.701423 9642 ts_tablet_manager.cc:1403] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 2.069s user 0.026s sys 0.031s
I20260503 13:58:30.653157 10530 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.679431 10310 raft_consensus.cc:515] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.747547 10547 raft_consensus.cc:493] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:30.697331 10537 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:30.697721 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:30.742172 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:30.748322 10310 leader_election.cc:290] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:30.749082 9524 raft_consensus.cc:2393] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
W20260503 13:58:30.703282 10419 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.614s user 0.007s sys 0.006s
I20260503 13:58:30.749733 9644 tablet_bootstrap.cc:654] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
W20260503 13:58:30.750765 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:30.746517 9519 pending_rounds.cc:85] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a: Aborting all ops after (but not including) 1
I20260503 13:58:30.751348 9519 pending_rounds.cc:107] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260503 13:58:30.751598 9519 raft_consensus.cc:2889] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:30.752795 9524 raft_consensus.cc:1275] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:30.753808 9524 raft_consensus.cc:1240] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:30.701870 10534 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:30.702795 10532 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:30.703886 10457 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:30.756011 9451 raft_consensus.cc:1275] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:30.758322 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:30.759919 9450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 356) took 2023 ms. Trace:
I20260503 13:58:30.705394 10541 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:30.760313 9450 rpcz_store.cc:276] 0503 13:58:28.736045 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:29.170858 (+434813us) service_pool.cc:224] Handling call
0503 13:58:30.758197 (+1587339us) spinlock_profiling.cc:243] Waited 6.72 ms on lock 0x7b5800000998. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46aff760 00007f9f46aff044 00007f9f4006c834 00007f9f4006c7c2
0503 13:58:30.759866 (+ 1669us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15446400}
I20260503 13:58:30.741308 10163 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.759495 9600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 338) took 2448 ms. Trace:
I20260503 13:58:30.759943 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 359) took 1482 ms. Trace:
W20260503 13:58:30.750710 10526 log.cc:927] Time spent T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.190s user 0.011s sys 0.022s
I20260503 13:58:30.761155 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:30.745433 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 361) took 1467 ms. Trace:
I20260503 13:58:30.763859 9978 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.762946 9451 rpcz_store.cc:276] 0503 13:58:29.277719 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.746535 (+1468816us) service_pool.cc:224] Handling call
0503 13:58:30.755760 (+ 9225us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:30.757518 (+ 1758us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:30.759896 (+ 2378us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:30.764524 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:30.762795 9600 rpcz_store.cc:276] 0503 13:58:28.310615 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.954509 (+643894us) service_pool.cc:224] Handling call
0503 13:58:29.003076 (+ 48567us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:29.021666 (+ 18590us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:30.759314 (+1737648us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b540031abd0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:30.759441 (+ 127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54325504}
I20260503 13:58:30.756170 10542 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.768119 9451 raft_consensus.cc:3060] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:30.756786 10179 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:30.770062 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 110780us
I20260503 13:58:30.756654 10193 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.748018 10547 raft_consensus.cc:515] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.696461 10535 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.684003 10503 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.768275 9978 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.776883 10017 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:30.767930 10541 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.761426 10287 raft_consensus.cc:493] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.701159 9444 rpcz_store.cc:276] 0503 13:58:28.771013 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.135760 (+1364747us) service_pool.cc:224] Handling call
0503 13:58:30.332680 (+196920us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:30.658232 (+325552us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:30.682576 (+ 24344us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:30.692543 10540 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.755481 10513 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.703598 10510 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.781127 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 410) took 2085 ms. Trace:
I20260503 13:58:30.748797 10537 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.782603 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:30.783385 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:30.783975 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:30.762173 10521 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.793823 9978 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:30.762260 10457 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
W20260503 13:58:30.795115 10518 log.cc:927] Time spent T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.153s user 0.000s sys 0.001s
I20260503 13:58:30.792464 9418 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:30.783069 10193 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.749781 9527 rpcz_store.cc:276] 0503 13:58:29.482582 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:29.517296 (+ 34714us) service_pool.cc:224] Handling call
0503 13:58:29.558768 (+ 41472us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:30.655892 (+1097124us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:30.682704 (+ 26812us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:30.797044 9453 rpcz_store.cc:276] 0503 13:58:28.696191 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:29.188114 (+491923us) service_pool.cc:224] Handling call
0503 13:58:29.213268 (+ 25154us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:29.213604 (+ 336us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:29.213631 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:29.214021 (+ 390us) log.cc:844] Serialized 26 byte log entry
0503 13:58:30.011094 (+797073us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:30.011179 (+ 85us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:30.011257 (+ 78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:30.780939 (+769682us) raft_consensus.cc:1685] finished
0503 13:58:30.780958 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:30.781069 (+ 111us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6312,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":938,"thread_start_us":774863,"threads_started":1,"wal-append.queue_time_us":1467088}
I20260503 13:58:30.798403 9601 raft_consensus.cc:2468] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:30.769663 10543 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.751314 10530 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.767645 9448 rpcz_store.cc:276] 0503 13:58:29.278275 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:29.630352 (+352077us) service_pool.cc:224] Handling call
0503 13:58:30.740746 (+1110394us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:30.779199 10453 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:30.797823 9600 raft_consensus.cc:2393] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:30.785089 10347 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.769361 10163 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.814096 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 406) took 2630 ms. Trace:
I20260503 13:58:30.849774 9526 rpcz_store.cc:276] 0503 13:58:28.184152 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:29.201410 (+1017258us) service_pool.cc:224] Handling call
0503 13:58:29.201717 (+ 307us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:29.202036 (+ 319us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:29.202062 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:29.202513 (+ 451us) log.cc:844] Serialized 52 byte log entry
0503 13:58:29.998976 (+796463us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:29.999016 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:29.999079 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:30.813905 (+814826us) raft_consensus.cc:1685] finished
0503 13:58:30.813923 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:30.814039 (+ 116us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1526,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":63,"thread_start_us":796120,"threads_started":1,"wal-append.queue_time_us":1187668}
I20260503 13:58:30.850800 9528 raft_consensus.cc:1275] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:30.802379 9519 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:30.770030 10507 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.802057 9418 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:30.792912 10396 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:30.793169 10251 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.877415 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:30.881374 9598 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:30.779706 10506 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:30.879524 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:30.800984 10028 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:30.797972 10544 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:30.759711 10534 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.760270 10532 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.808974 10170 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:30.901311 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:30.800454 10463 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.883522 9601 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
W20260503 13:58:30.836024 9416 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.26.195:39997 timed out after 1.840s (SENT) [suppressed 1 similar messages]
I20260503 13:58:30.795737 10017 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.781314 10179 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:30.886854 10533 log.cc:927] Time spent T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.238s user 0.001s sys 0.001s
I20260503 13:58:30.928334 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:30.934258 9601 raft_consensus.cc:3060] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:30.876981 10113 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:30.896951 9567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:30.801514 10287 raft_consensus.cc:515] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.943833 9598 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:30.966612 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:30.883002 10545 raft_consensus.cc:2804] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:30.967361 9598 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:30.970208 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:30.880750 10347 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.801326 10513 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.780237 10542 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:30.927594 10531 log.cc:927] Time spent T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.229s user 0.001s sys 0.000s
I20260503 13:58:30.936635 10511 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:30.929580 10445 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:30.978158 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:30.887910 10251 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.983023 9601 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:30.984017 10295 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.984807 10431 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.876766 10453 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.813002 10521 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.886873 10543 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.886416 10507 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.966513 10113 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.895470 10544 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.896322 10506 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.927892 10463 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:31.015116 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:30.989087 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:31.289268 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 487229us
I20260503 13:58:31.480973 9598 raft_consensus.cc:2468] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
W20260503 13:58:31.289430 9520 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.004s sys 0.004s
I20260503 13:58:31.025521 9451 raft_consensus.cc:1275] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:31.337607 9566 leader_election.cc:304] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:31.482738 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:31.482456 9418 consensus_peers.cc:597] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 -> Peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Couldn't send request to peer 428b342dc4f041df956015e7d7cd173a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260503 13:58:31.065435 9599 raft_consensus.cc:1275] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260503 13:58:31.483198 9520 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.721s user 0.006s sys 0.010s
W20260503 13:58:31.288185 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 452144us
W20260503 13:58:31.484539 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:31.485399 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 365) took 2206 ms. Trace:
I20260503 13:58:31.485790 9451 rpcz_store.cc:276] 0503 13:58:29.279166 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.767874 (+1488708us) service_pool.cc:224] Handling call
0503 13:58:30.768049 (+ 175us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:31.485166 (+717117us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:31.485349 (+ 183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:31.489260 9642 raft_consensus.cc:359] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:30.968333 10545 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:31.491137 9642 raft_consensus.cc:385] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260503 13:58:31.491406 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:31.494664 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:31.134845 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 329) took 2591 ms. Trace:
W20260503 13:58:31.491994 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 154489us
W20260503 13:58:31.492385 9491 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.000s sys 0.005s
W20260503 13:58:31.481460 9524 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.021s sys 0.017s
W20260503 13:58:31.495316 9491 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.520s user 0.001s sys 0.005s
W20260503 13:58:31.495328 9524 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.527s user 0.021s sys 0.017s
I20260503 13:58:31.495080 9597 rpcz_store.cc:276] 0503 13:58:28.543185 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.998658 (+455473us) service_pool.cc:224] Handling call
0503 13:58:29.002869 (+ 4211us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:29.997920 (+995051us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:29.997950 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:30.073887 (+ 75937us) log.cc:844] Serialized 26 byte log entry
0503 13:58:30.604027 (+530140us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:30.604118 (+ 91us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:30.604187 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:31.134594 (+530407us) raft_consensus.cc:1685] finished
0503 13:58:31.134614 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:31.134787 (+ 173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":205819,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":65,"thread_start_us":1339219,"threads_started":2,"wal-append.queue_time_us":503814}
I20260503 13:58:31.496914 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:30.933077 10170 raft_consensus.cc:2804] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260503 13:58:31.488976 10419 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.725s user 0.001s sys 0.018s
W20260503 13:58:31.501632 10419 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s user 0.001s sys 0.018s
I20260503 13:58:31.504527 9418 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:31.494345 9642 raft_consensus.cc:740] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:31.065225 10028 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
W20260503 13:58:31.134048 10555 log.cc:927] Time spent T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20260503 13:58:31.026765 10179 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:30.989346 9595 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 321) took 3016 ms (client timeout 2543 ms). Trace:
W20260503 13:58:31.478770 10550 log.cc:927] Time spent T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.373s user 0.000s sys 0.000s
I20260503 13:58:31.507540 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 327) took 3543 ms. Trace:
I20260503 13:58:31.508807 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:31.507026 9642 consensus_queue.cc:260] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:31.461902 10500 log.cc:927] Time spent T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 1.470s user 0.004s sys 0.011s
W20260503 13:58:31.330000 10358 consensus_meta.cc:287] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 1.727s user 0.010s sys 0.001s
I20260503 13:58:31.511308 10551 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:31.382941 10537 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:30.980756 10361 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:31.533185 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:31.508860 10452 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:31.512658 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:31.027427 10523 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:31.536360 10561 raft_consensus.cc:2749] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:31.027577 10552 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:31.490132 10396 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:30.989187 10542 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:31.490635 10193 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:31.071197 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 416) took 1507 ms. Trace:
I20260503 13:58:31.538336 9594 rpcz_store.cc:276] 0503 13:58:27.963615 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.079750 (+116135us) service_pool.cc:224] Handling call
0503 13:58:28.999261 (+919511us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:29.647427 (+648166us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:31.461866 (+1814439us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b54002b98d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:31.480319 (+ 18453us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b54002b98d0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:31.504400 (+ 24081us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3544955008}
I20260503 13:58:31.434309 10510 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:31.537541 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:31.026193 10554 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:31.570199 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:31.555464 10347 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:31.461719 10557 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:30.990068 10434 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:31.490876 10310 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:31.319804 10556 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:31.493245 10295 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:31.505125 10170 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:31.563125 9600 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:31.357784 10457 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:31.557796 10511 raft_consensus.cc:2804] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:31.506239 10545 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:31.563555 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:31.564584 9418 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:31.508280 10017 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:31.566068 9597 raft_consensus.cc:2468] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:31.026713 10163 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:31.572717 9601 raft_consensus.cc:2468] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
W20260503 13:58:31.531993 9595 rpcz_store.cc:269] 0503 13:58:27.963103 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.049049 (+ 85946us) service_pool.cc:224] Handling call
0503 13:58:30.969677 (+2920628us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b54002616d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:30.979681 (+ 10004us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1147359104}
I20260503 13:58:31.508136 10028 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:31.561424 10287 leader_election.cc:290] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:31.560647 9527 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.017s sys 0.034s
I20260503 13:58:31.562204 10358 raft_consensus.cc:515] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:31.570560 10554 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:31.571237 10557 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:31.493413 10431 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:31.976815 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 344) took 1013 ms. Trace:
I20260503 13:58:31.795540 10532 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:31.977278 9597 rpcz_store.cc:276] 0503 13:58:30.963834 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:31.510708 (+546874us) service_pool.cc:224] Handling call
0503 13:58:31.976754 (+466046us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:31.974835 9642 ts_tablet_manager.cc:1434] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 1.228s user 0.007s sys 0.015s
I20260503 13:58:31.977658 9601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 342) took 1225 ms. Trace:
I20260503 13:58:31.979355 9601 rpcz_store.cc:276] 0503 13:58:30.752629 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:31.497745 (+745116us) service_pool.cc:224] Handling call
0503 13:58:31.977607 (+479862us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:31.769546 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 263855us
W20260503 13:58:31.980751 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:31.932312 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:31.983188 9600 raft_consensus.cc:2393] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:31.983757 9598 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:31.984099 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 345) took 1006 ms. Trace:
I20260503 13:58:31.984323 9598 rpcz_store.cc:276] 0503 13:58:30.978032 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:31.538634 (+560602us) service_pool.cc:224] Handling call
0503 13:58:31.984060 (+445426us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:31.928231 9644 tablet_bootstrap.cc:492] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
W20260503 13:58:31.880738 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 367852us
W20260503 13:58:31.932626 9567 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.968s user 0.000s sys 0.011s
W20260503 13:58:31.987116 9567 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.022s user 0.000s sys 0.013s
I20260503 13:58:31.986682 9644 ts_tablet_manager.cc:1403] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 2.666s user 0.036s sys 0.045s
I20260503 13:58:31.572053 10434 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:31.571688 10506 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:31.922350 9596 raft_consensus.cc:3055] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:31.569684 9448 rpcz_store.cc:276] 0503 13:58:29.561781 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.880643 (+1318862us) service_pool.cc:224] Handling call
0503 13:58:30.968968 (+ 88325us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:30.979229 (+ 10261us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:30.989431 (+ 10202us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:30.998389 (+ 8958us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:31.027233 (+ 28844us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:31.027464 (+ 231us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:31.069252 (+ 41788us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:31.970306 10243 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:32.061308 9491 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:31.920971 9447 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 349) took 3305 ms (client timeout 2343 ms). Trace:
I20260503 13:58:31.772001 10453 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.039973 9978 consensus_queue.cc:1048] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260503 13:58:31.988868 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 424357us
I20260503 13:58:31.767966 10560 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:31.977761 9527 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.953s user 0.027s sys 0.044s
I20260503 13:58:31.723026 10541 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:31.979172 9642 ts_tablet_manager.cc:1467] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a: Tablet startup took 1232ms
W20260503 13:58:32.063735 10553 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.742s user 0.004s sys 0.018s
I20260503 13:58:31.565245 10540 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:31.606033 10511 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:31.582746 10170 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:32.092831 10511 raft_consensus.cc:3060] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:31.795317 9602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 332) took 3230 ms. Trace:
W20260503 13:58:32.057852 9525 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.990s user 0.006s sys 0.013s
I20260503 13:58:32.059149 9567 leader_election.cc:304] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:31.922526 10513 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:31.569033 10523 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:32.097105 9525 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.030s user 0.006s sys 0.013s
I20260503 13:58:31.721756 10457 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:31.705403 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.061312 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:31.580490 10396 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.099195 9598 raft_consensus.cc:2468] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:31.977850 10459 raft_consensus.cc:493] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:31.922031 10295 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.063424 10372 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:32.063015 10251 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:32.095546 10553 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.774s user 0.005s sys 0.019s
W20260503 13:58:31.830657 10528 log.cc:927] Time spent T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 1.176s user 0.003s sys 0.007s
I20260503 13:58:31.572305 10452 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.075639 10463 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:32.064406 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:31.675124 10510 consensus_queue.cc:1048] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.386s
W20260503 13:58:31.769382 10500 log.cc:927] Time spent T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20260503 13:58:32.113912 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:32.067610 10577 raft_consensus.cc:493] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:31.570508 10543 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.045795 9596 raft_consensus.cc:740] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:31.675740 10556 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:31.628767 10563 log.cc:927] Time spent T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
W20260503 13:58:32.092196 9642 ts_tablet_manager.cc:1469] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a: Trace:
0503 13:58:05.614342 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:05.968707 (+354365us) service_pool.cc:224] Handling call
0503 13:58:05.970343 (+ 1636us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:05.970430 (+ 87us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.042212 (+ 71782us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.190111 (+147899us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.190498 (+ 387us) inbound_call.cc:177] Queueing success response
0503 13:58:28.623639 (+22433141us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0503 13:58:30.746806 (+2123167us) ts_tablet_manager.cc:1435] Starting tablet replica
0503 13:58:31.027819 (+281013us) tablet_replica.cc:261] Starting instrumentation
0503 13:58:31.028266 (+ 447us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":225227,"raft.queue_time_us":222304,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":66,"tablet-open.queue_time_us":22433081,"thread_start_us":223645,"threads_started":1}
I20260503 13:58:32.057178 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:31.603726 10310 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:31.894788 10549 log.cc:927] Time spent T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.576s user 0.001s sys 0.000s
I20260503 13:58:31.985678 10451 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:32.113062 10513 raft_consensus.cc:2749] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260503 13:58:32.062865 9491 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:32.046555 10193 consensus_queue.cc:1048] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260503 13:58:31.920140 10521 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.042203 10283 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:32.110981 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 51923.5us
I20260503 13:58:32.111485 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.083351 9978 consensus_queue.cc:1059] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260503 13:58:32.111456 10372 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:32.114321 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:32.104089 9566 connection.cc:254] RPC call timeout handler was delayed by -1.1426900889999843s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
I20260503 13:58:32.044780 10434 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:31.572676 10551 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.133301 10589 consensus_queue.cc:1048] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260503 13:58:32.065923 10243 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.101150 10113 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.047224 10017 consensus_queue.cc:1048] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20260503 13:58:32.109252 10537 raft_consensus.cc:2804] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:32.109485 10287 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:32.111734 10295 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:32.149645 9642 tablet_bootstrap.cc:492] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:31.707031 10530 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:32.064858 9447 rpcz_store.cc:269] 0503 13:58:28.615828 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.799460 (+183632us) service_pool.cc:224] Handling call
0503 13:58:31.920708 (+3121248us) spinlock_profiling.cc:243] Waited 384 ms on lock 0x7b5400421dd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:31.920861 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":883306368}
I20260503 13:58:31.607254 10552 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.158141 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 347) took 3554 ms. Trace:
I20260503 13:58:32.112855 10507 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.173432 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.140203 9595 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:32.090847 10560 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.164304 10523 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.202564 10564 consensus_queue.cc:1048] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260503 13:58:32.106084 10457 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:32.210851 9446 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 369) took 2822 ms (client timeout 1877 ms). Trace:
W20260503 13:58:32.211242 9446 rpcz_store.cc:269] 0503 13:58:29.388350 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:29.517517 (+129167us) service_pool.cc:224] Handling call
0503 13:58:32.210640 (+2693123us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b5400303254. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:32.210785 (+ 145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2287599872}
I20260503 13:58:32.211985 10457 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.203311 10310 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.178484 10521 consensus_queue.cc:1048] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.039s
I20260503 13:58:32.202360 10358 leader_election.cc:290] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.193709 10028 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.216246 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.217237 10287 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:32.164834 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 72563.1us
I20260503 13:58:32.168197 9978 consensus_queue.cc:1243] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer 1cb63886040047559d3893923cc9aaf6 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260503 13:58:32.139810 10431 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.150148 10544 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.172011 9567 leader_election.cc:304] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
W20260503 13:58:32.240069 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 68133.9us
I20260503 13:58:32.240926 9978 raft_consensus.cc:2804] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:32.241350 9978 raft_consensus.cc:697] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:32.242030 9978 consensus_queue.cc:237] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.162359 9491 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:32.197618 9449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 415) took 3401 ms. Trace:
I20260503 13:58:32.197731 9445 rpcz_store.cc:276] 0503 13:58:28.597497 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.597601 (+ 104us) service_pool.cc:224] Handling call
0503 13:58:28.610391 (+ 12790us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:28.610579 (+ 188us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:28.610601 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:28.610945 (+ 344us) log.cc:844] Serialized 26 byte log entry
0503 13:58:29.145683 (+534738us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:29.162825 (+ 17142us) log.cc:844] Serialized 14 byte log entry
0503 13:58:29.184948 (+ 22123us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:29.185023 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:30.988882 (+1803859us) spinlock_profiling.cc:243] Waited 4.16 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:32.151193 (+1162311us) raft_consensus.cc:1685] finished
0503 13:58:32.151272 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:32.151408 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21857,"prepare.queue_time_us":24387,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":22937,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":4414,"spinlock_wait_cycles":9573760,"thread_start_us":448880,"threads_started":1,"wal-append.queue_time_us":827349}
I20260503 13:58:32.217483 9598 raft_consensus.cc:2468] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:32.109444 10459 raft_consensus.cc:515] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:32.178009 10566 log.cc:927] Time spent T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.373s user 0.001s sys 0.000s
I20260503 13:58:32.197129 10554 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.243773 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:32.202134 10499 log.cc:927] Time spent T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 2.557s user 0.019s sys 0.011s
W20260503 13:58:32.246592 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 84311.2us
I20260503 13:58:32.212322 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.221284 10238 raft_consensus.cc:493] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:32.263796 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:32.251652 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260503 13:58:32.193943 10571 log.cc:927] Time spent T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260503 13:58:32.196342 10556 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.157056 10451 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:32.174043 10253 consensus_queue.cc:1048] T 54bf4ee675fb4fcf800b317817906dcc P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.081s
I20260503 13:58:32.243009 9449 rpcz_store.cc:276] 0503 13:58:28.796172 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.665397 (+1869225us) service_pool.cc:224] Handling call
0503 13:58:30.683758 (+ 18361us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:30.877977 (+194219us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:30.878008 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:30.878626 (+ 618us) log.cc:844] Serialized 52 byte log entry
0503 13:58:31.480465 (+601839us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:31.931717 (+451252us) log.cc:844] Serialized 14 byte log entry
0503 13:58:32.063301 (+131584us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.063520 (+ 219us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:32.197433 (+133913us) raft_consensus.cc:1685] finished
0503 13:58:32.197450 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:32.197566 (+ 116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6649,"prepare.queue_time_us":583314,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":2523,"raft.queue_time_us":186402,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":47369,"spinlock_wait_cycles":17172480,"thread_start_us":1187812,"threads_started":4,"wal-append.queue_time_us":613180}
I20260503 13:58:32.223724 10537 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:32.134768 10577 raft_consensus.cc:515] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.237257 10310 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:32.240341 10510 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:32.254812 10544 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:32.217976 10295 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.267910 9598 raft_consensus.cc:1240] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:32.167580 10283 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.254604 10358 raft_consensus.cc:2804] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:32.258496 9491 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:32.296120 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.262162 10530 raft_consensus.cc:493] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:32.278826 9527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 375) took 1622 ms. Trace:
I20260503 13:58:32.310719 10577 leader_election.cc:290] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.309753 9527 rpcz_store.cc:276] 0503 13:58:30.654784 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.876624 (+221840us) service_pool.cc:224] Handling call
0503 13:58:30.932491 (+ 55867us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:30.969298 (+ 36807us) raft_consensus.cc:1534] Early marking committed up to index 1
0503 13:58:32.098642 (+1129344us) log.cc:844] Serialized 14 byte log entry
0503 13:58:32.247205 (+148563us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:32.255023 (+ 7818us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:32.256990 (+ 1967us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.265187 (+ 8197us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:32.276908 (+ 11721us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1249,"prepare.queue_time_us":803906,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":8010,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":8251,"thread_start_us":1188922,"threads_started":2}
I20260503 13:58:32.311190 10583 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:32.265542 10557 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.266743 10238 raft_consensus.cc:515] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.314222 10238 leader_election.cc:290] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.277073 10537 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:32.265914 10564 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:32.272725 10594 consensus_queue.cc:1059] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20260503 13:58:32.303714 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 341) took 1551 ms. Trace:
W20260503 13:58:32.306564 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 48138us
I20260503 13:58:32.267067 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.276990 10028 consensus_queue.cc:1048] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
F20260503 13:58:32.309367 8299 test_workload.cc:382] Timed out: Timed out waiting for Table Creation
I20260503 13:58:32.266877 9446 raft_consensus.cc:2468] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:32.416754 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.414340 9493 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:37228 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 445)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 458)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 440)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 446)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 462)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 437)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 454)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 441)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 451)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 460)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 442)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 461)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 435)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 436)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 455)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 456)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 457)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 459)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 378)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 379)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 382)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 385)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 389)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 390)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 394)
I20260503 13:58:32.418352 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.311626 10583 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.421231 10583 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.418673 9493 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:37228 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 445)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 458)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 440)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 446)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 462)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 437)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 454)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 441)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 451)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 460)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 442)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 461)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 435)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 436)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 455)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 456)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 457)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 459)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 378)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 379)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 382)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 385)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 389)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 390)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 394)
W20260503 13:58:32.299938 9453 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.018s sys 0.033s
I20260503 13:58:32.268967 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:32.422187 9453 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.669s user 0.018s sys 0.036s
I20260503 13:58:32.311786 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.269143 9594 raft_consensus.cc:2468] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:32.423591 10547 leader_election.cc:290] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.267553 9448 raft_consensus.cc:1240] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
*** Check failure stack trace: ***
*** Aborted at 1777816712 (unix time) try "date -d @1777816712" if you are using GNU date ***
I20260503 13:58:32.322052 9491 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:32.321282 9598 rpcz_store.cc:276] 0503 13:58:30.752376 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:32.252898 (+1500522us) service_pool.cc:224] Handling call
0503 13:58:32.267749 (+ 14851us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:32.303480 (+ 35731us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.303663 (+ 183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:32.323694 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 368) took 1670 ms. Trace:
I20260503 13:58:32.424646 9528 rpcz_store.cc:276] 0503 13:58:30.653141 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.749168 (+ 96027us) service_pool.cc:224] Handling call
0503 13:58:30.749452 (+ 284us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:30.851173 (+101721us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.323466 (+1472293us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54003fd754. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:32.323637 (+ 171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":232283008}
I20260503 13:58:32.425071 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.425594 9598 raft_consensus.cc:2468] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:32.425683 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.308214 10358 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:32.277281 10470 consensus_queue.cc:1048] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260503 13:58:32.425923 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.426199 10358 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:32.426348 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.426512 9594 raft_consensus.cc:2468] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:32.315254 10251 raft_consensus.cc:2804] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:32.427306 10251 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:32.427379 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.427728 10251 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:32.096959 9602 rpcz_store.cc:276] 0503 13:58:28.565085 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:29.077423 (+512338us) service_pool.cc:224] Handling call
0503 13:58:29.077802 (+ 379us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:30.660501 (+1582699us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:30.662963 (+ 2462us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:30.695087 (+ 32124us) log.cc:844] Serialized 26 byte log entry
0503 13:58:30.933811 (+238724us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:30.964403 (+ 30592us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:30.968644 (+ 4241us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:31.706735 (+738091us) raft_consensus.cc:1685] finished
0503 13:58:31.721703 (+ 14968us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:31.778279 (+ 56576us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":130988,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":64,"thread_start_us":1772135,"threads_started":2,"wal-append.queue_time_us":277659}
I20260503 13:58:32.346544 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:32.350147 10582 log.cc:927] Time spent T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260503 13:58:32.299456 10419 consensus_queue.cc:1048] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260503 13:58:32.351617 10503 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.317420 10452 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.317893 10589 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:32.295372 10310 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.430820 10589 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.428934 9493 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:37228 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 458)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 440)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 446)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 462)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 437)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 454)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 441)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 451)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 460)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 442)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 461)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 435)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 436)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 455)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 456)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 457)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 459)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 378)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 379)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 382)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 385)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 389)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 390)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 394)
I20260503 13:58:32.432049 10310 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:32.356081 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 100719us
I20260503 13:58:32.432652 10589 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.432914 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.434015 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:32.434633 9566 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:32.435189 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:32.359308 10508 log.cc:927] Time spent T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 2.166s user 0.004s sys 0.015s
I20260503 13:58:32.295576 10556 consensus_queue.cc:1048] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260503 13:58:32.299767 10510 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.433588 9493 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:37220 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 440)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 446)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 462)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 437)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 454)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 441)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 451)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 460)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 442)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 461)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 435)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 436)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 455)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 457)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 459)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 378)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 379)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 382)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 385)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 389)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 390)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 394)
I20260503 13:58:32.364866 10170 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.365366 10287 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.366879 10451 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:32.362450 9519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 374) took 1707 ms. Trace:
I20260503 13:58:32.441735 9519 rpcz_store.cc:276] 0503 13:58:30.654565 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.757657 (+103092us) service_pool.cc:224] Handling call
0503 13:58:30.757882 (+ 225us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:30.966348 (+208466us) raft_consensus.cc:1534] Early marking committed up to index 1
0503 13:58:30.966377 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:30.966418 (+ 41us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:30.966451 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.362223 (+1395772us) spinlock_profiling.cc:243] Waited 44.4 ms on lock 0x7b54002067d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:32.362271 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:32.362392 (+ 121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":219364,"raft.queue_time_us":95747,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":60593,"spinlock_wait_cycles":102053120}
I20260503 13:58:32.443202 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.443888 9519 raft_consensus.cc:2393] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
W20260503 13:58:32.444007 10522 log.cc:927] Time spent T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 1.784s user 0.013s sys 0.000s
I20260503 13:58:32.303944 10544 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.345626 10562 raft_consensus.cc:3055] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:32.371008 10552 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.317201 10530 raft_consensus.cc:515] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.446985 10170 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.447602 10510 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.448642 10530 leader_election.cc:290] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.370771 10545 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.450464 10545 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.452339 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 371) took 2973 ms. Trace:
I20260503 13:58:32.435703 10580 raft_consensus.cc:2804] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:32.436630 10547 raft_consensus.cc:2804] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:32.448916 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.452764 9453 rpcz_store.cc:276] 0503 13:58:29.479029 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.811421 (+1332392us) service_pool.cc:224] Handling call
0503 13:58:30.880231 (+ 68810us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:31.533873 (+653642us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:31.728409 (+194536us) log.cc:844] Serialized 14 byte log entry
0503 13:58:32.422577 (+694168us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:32.422624 (+ 47us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:32.422661 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.452119 (+ 29458us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b540007fdd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:32.452164 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:32.452283 (+ 119us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":576138,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":1627,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":3911,"spinlock_wait_cycles":1502769024,"thread_start_us":839579,"threads_started":2,"wal-append.queue_time_us":619466}
I20260503 13:58:32.454492 9601 raft_consensus.cc:1240] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:32.372888 9452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 405) took 3909 ms. Trace:
W20260503 13:58:32.455479 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:32.444962 10184 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.373497 9450 raft_consensus.cc:1217] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2] Dedup: 3.2->[]
I20260503 13:58:32.369724 10459 leader_election.cc:290] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.455024 9601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 345) took 1701 ms. Trace:
I20260503 13:58:32.323813 10564 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.374900 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 430) took 1596 ms. Trace:
W20260503 13:58:32.376271 10578 log.cc:927] Time spent T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260503 13:58:32.456439 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 420) took 1662 ms. Trace:
I20260503 13:58:32.362288 10283 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.378892 10295 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.460939 9598 raft_consensus.cc:1240] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:32.383963 9643 tablet_bootstrap.cc:492] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:32.384083 9596 consensus_queue.cc:260] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.385816 10511 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.386922 10372 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:32.388317 9417 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46714 dropped due to backpressure: service queue is full with 50 items [suppressed 1 similar messages]
I20260503 13:58:32.459986 9525 rpcz_store.cc:276] 0503 13:58:30.778391 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.978030 (+199639us) service_pool.cc:224] Handling call
0503 13:58:30.978291 (+ 261us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:30.978571 (+ 280us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:30.978595 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:30.978985 (+ 390us) log.cc:844] Serialized 26 byte log entry
0503 13:58:32.097554 (+1118569us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:32.097594 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.097661 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:32.374718 (+277057us) raft_consensus.cc:1685] finished
0503 13:58:32.374737 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:32.374849 (+ 112us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":341197,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":1029875,"threads_started":1,"wal-append.queue_time_us":1374842}
W20260503 13:58:32.389014 10517 log.cc:927] Time spent T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 1.738s user 0.000s sys 0.014s
I20260503 13:58:32.380304 10243 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.463981 9523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 357) took 3750 ms. Trace:
I20260503 13:58:32.465195 9596 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:32.464327 9523 rpcz_store.cc:276] 0503 13:58:28.713628 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:29.087336 (+373708us) service_pool.cc:224] Handling call
0503 13:58:29.087704 (+ 368us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:29.088090 (+ 386us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:29.088121 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:29.088632 (+ 511us) log.cc:844] Serialized 26 byte log entry
0503 13:58:29.581546 (+492914us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:29.599524 (+ 17978us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:29.599619 (+ 95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:32.388217 (+2788598us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:32.463713 (+ 75496us) raft_consensus.cc:1685] finished
0503 13:58:32.463737 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:32.463924 (+ 187us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1348,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"spinlock_wait_cycles":2779392,"thread_start_us":463697,"threads_started":1,"wal-append.queue_time_us":1561722}
I20260503 13:58:32.390499 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:32.466066 10511 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.393873 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 323) took 3910 ms. Trace:
I20260503 13:58:32.466890 9449 raft_consensus.cc:2368] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4.
W20260503 13:58:32.396898 10529 log.cc:927] Time spent T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 1.742s user 0.015s sys 0.027s
W20260503 13:58:32.396795 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 424697us
I20260503 13:58:32.466162 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.467834 9523 raft_consensus.cc:2468] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:32.400663 9642 tablet_bootstrap.cc:654] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:32.401400 9524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 372) took 1747 ms. Trace:
I20260503 13:58:32.402251 9522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 410) took 3686 ms. Trace:
I20260503 13:58:32.468688 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.395161 10537 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.468928 9522 rpcz_store.cc:276] 0503 13:58:28.715670 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:29.109412 (+393742us) service_pool.cc:224] Handling call
0503 13:58:29.646122 (+536710us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:29.646717 (+ 595us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.402047 (+2755330us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b54001554d0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:32.402177 (+ 130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1233597696}
I20260503 13:58:32.466567 9599 rpcz_store.cc:276] 0503 13:58:28.483175 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.956990 (+473815us) service_pool.cc:224] Handling call
0503 13:58:28.957259 (+ 269us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:31.484154 (+2526895us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.393692 (+909538us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b5400060ed4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:32.393827 (+ 135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4847541504}
I20260503 13:58:32.412091 10434 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.423141 9527 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:32.279369 10543 consensus_queue.cc:1048] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
W20260503 13:58:32.429308 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 107324us
I20260503 13:58:32.358991 10551 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.435679 9595 raft_consensus.cc:3060] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:32.457346 10562 raft_consensus.cc:740] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:32.436111 10199 leader_election.cc:290] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.434515 9602 raft_consensus.cc:3060] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:32.439818 9445 raft_consensus.cc:2468] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:32.439350 9493 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:37228 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 440)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 446)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 462)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 437)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 402)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 454)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 441)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 451)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 460)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 442)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 461)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 435)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 436)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 455)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 457)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 459)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 378)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 379)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 382)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 385)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 389)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 390)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 394)
I20260503 13:58:32.441092 9521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 424) took 1833 ms. Trace:
I20260503 13:58:32.446743 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.457561 9450 raft_consensus.cc:1240] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:32.459197 9601 rpcz_store.cc:276] 0503 13:58:30.753306 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:32.454077 (+1700771us) service_pool.cc:224] Handling call
0503 13:58:32.454341 (+ 264us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:32.454831 (+ 490us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.454980 (+ 149us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:32.429378 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.382973 9593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 331) took 3825 ms. Trace:
I20260503 13:58:32.509404 10564 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.462428 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 344) took 1709 ms. Trace:
I20260503 13:58:32.462443 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.459064 10547 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:32.427654 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.452930 10580 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:32.525262 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:32.525910 9525 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:32.444641 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.471863 9524 rpcz_store.cc:276] 0503 13:58:30.654088 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.754484 (+100396us) service_pool.cc:224] Handling call
0503 13:58:30.754734 (+ 250us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:30.937373 (+182639us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:31.495890 (+558517us) log.cc:844] Serialized 14 byte log entry
0503 13:58:31.761541 (+265651us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:31.761597 (+ 56us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:31.761638 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.401193 (+639555us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b540004ea54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:32.401237 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:32.401350 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47087,"prepare.queue_time_us":738612,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":48731,"raft.run_cpu_time_us":53,"raft.run_wall_time_us":46,"spinlock_wait_cycles":419598592,"thread_start_us":744288,"threads_started":2,"wal-append.queue_time_us":563825}
I20260503 13:58:32.526921 9519 raft_consensus.cc:2468] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:32.527240 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.469269 9523 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:32.528031 9524 raft_consensus.cc:2410] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260503 13:58:32.470171 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.478574 10537 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.460763 9448 rpcz_store.cc:276] 0503 13:58:30.793495 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:32.075757 (+1282262us) service_pool.cc:224] Handling call
0503 13:58:32.134260 (+ 58503us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:32.424349 (+290089us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.456207 (+ 31858us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54004213d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:32.456380 (+ 173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":270092544}
I20260503 13:58:32.483184 10562 consensus_queue.cc:260] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.474099 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.485702 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.487142 9521 rpcz_store.cc:276] 0503 13:58:30.607509 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.607649 (+ 140us) service_pool.cc:224] Handling call
0503 13:58:30.607981 (+ 332us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:30.703559 (+ 95578us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.440905 (+1737346us) spinlock_profiling.cc:243] Waited 83.9 ms on lock 0x7b540004e054. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:32.441042 (+ 137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":192861440}
I20260503 13:58:32.488240 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.437909 9419 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:46726 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 445)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 441)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 393)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 394)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 428)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 442)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 398)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 429)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 447)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 440)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 436)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 433)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 435)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 451)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 425)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 449)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 422)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 424)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 427)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 374)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 446)
I20260503 13:58:32.508251 10251 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.461699 9643 ts_tablet_manager.cc:1403] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 3.258s user 0.038s sys 0.068s
I20260503 13:58:32.510052 9598 rpcz_store.cc:276] 0503 13:58:30.753094 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:32.441664 (+1688570us) service_pool.cc:224] Handling call
0503 13:58:32.460783 (+ 19119us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:32.462243 (+ 1460us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.462390 (+ 147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:32.461367 10358 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.508984 9593 rpcz_store.cc:276] 0503 13:58:28.557463 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:29.030847 (+473384us) service_pool.cc:224] Handling call
0503 13:58:29.451167 (+420320us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:29.469139 (+ 17972us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.382767 (+2913628us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b54002594d0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:32.382916 (+ 149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":817154432}
W20260503 13:58:32.624807 9567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:32.519891 10547 raft_consensus.cc:3060] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
W20260503 13:58:32.521064 9494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:32.593127 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.653595 9524 raft_consensus.cc:2468] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:32.654165 10251 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.654239 10543 raft_consensus.cc:493] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:32.654424 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.591394 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.654817 10543 raft_consensus.cc:515] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.655244 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.652922 9600 raft_consensus.cc:2393] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:32.655619 9597 raft_consensus.cc:2468] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:32.657342 10543 leader_election.cc:290] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.661469 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.532913 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.651945 10562 raft_consensus.cc:3060] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:32.662444 9519 raft_consensus.cc:2468] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:32.662664 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.663872 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.663993 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.664459 9597 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:32.665423 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.666033 9597 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:32.528474 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.615059 9525 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:32.667352 9601 raft_consensus.cc:2393] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:32.668278 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:32.669512 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:32.669349 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.672617 9525 raft_consensus.cc:2393] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:32.631081 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:32.675007 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.673466 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.652516 9527 raft_consensus.cc:2410] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260503 13:58:32.580266 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:32.655376 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:32.670820 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:32.674919 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.678093 9524 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:32.665725 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:32.655845 9598 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:32.665096 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.676827 9519 raft_consensus.cc:2468] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:32.675941 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:32.677834 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:32.679533 9528 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:32.679842 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.678813 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 419) took 1885 ms. Trace:
I20260503 13:58:32.680260 9521 raft_consensus.cc:3060] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:32.680697 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.681265 9519 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:32.680938 9451 rpcz_store.cc:276] 0503 13:58:30.793276 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:31.500985 (+707709us) service_pool.cc:224] Handling call
0503 13:58:31.501196 (+ 211us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:32.153985 (+652789us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:32.154017 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:32.154057 (+ 40us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:32.154086 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.678584 (+524498us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b54002e54d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:32.678674 (+ 90us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:32.678768 (+ 94us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1500857088}
I20260503 13:58:32.667110 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:32.682986 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.683560 9593 raft_consensus.cc:2468] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:32.683982 9567 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:32.684346 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.685374 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.682986 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.686293 9451 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:32.687028 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.687399 9567 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:32.455051 9452 rpcz_store.cc:276] 0503 13:58:28.463024 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.464480 (+ 1456us) service_pool.cc:224] Handling call
0503 13:58:28.520648 (+ 56168us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:28.564842 (+ 44194us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:28.565820 (+ 978us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:28.616043 (+ 50223us) log.cc:844] Serialized 52 byte log entry
0503 13:58:29.252712 (+636669us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:29.253221 (+ 509us) log.cc:844] Serialized 14 byte log entry
0503 13:58:29.286834 (+ 33613us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:29.286923 (+ 89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:32.112914 (+2825991us) spinlock_profiling.cc:243] Waited 8.62 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:32.372571 (+259657us) raft_consensus.cc:1685] finished
0503 13:58:32.372675 (+ 104us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:32.372829 (+ 154us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":134,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":3028,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":8754,"spinlock_wait_cycles":19828480,"thread_start_us":552151,"threads_started":1,"wal-append.queue_time_us":973809}
I20260503 13:58:32.687969 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.688393 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.688803 9567 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:32.689086 9452 raft_consensus.cc:2368] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4.
I20260503 13:58:32.689306 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.685632 10459 raft_consensus.cc:2804] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:32.689837 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.690339 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.690955 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:32.691687 9452 raft_consensus.cc:2393] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 5: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:32.688751 9451 raft_consensus.cc:2393] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:32.668898 10543 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:32.691824 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.691130 9527 raft_consensus.cc:2410] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 4 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260503 13:58:32.692847 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.529515 10580 raft_consensus.cc:3060] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:32.694068 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:32.682066 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.695216 9567 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:32.681430 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.696487 9602 raft_consensus.cc:2468] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:32.699199 9451 raft_consensus.cc:2410] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260503 13:58:32.630620 10396 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.692426 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.659299 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.461916 9417 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46714 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 445)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 441)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 393)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 394)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 428)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 442)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 398)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 429)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 447)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 440)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 436)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 433)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 435)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 451)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 425)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 449)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 422)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 424)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 427)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 374)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 446)
I20260503 13:58:32.692715 10506 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:32.699375 9600 raft_consensus.cc:2468] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:32.798609 9524 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:32.761965 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.842196 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 429) took 2064 ms. Trace:
I20260503 13:58:32.768936 10243 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:32.688733 10347 raft_consensus.cc:2804] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:32.699200 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.845136 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.774865 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.689850 10459 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:32.848361 10459 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:32.848453 9644 raft_consensus.cc:359] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.848997 10336 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
W20260503 13:58:32.837862 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:32.849189 9644 raft_consensus.cc:385] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:32.849989 9644 raft_consensus.cc:740] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:32.836282 10506 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.673166 10179 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:32.847764 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 167557us
I20260503 13:58:32.794600 9521 raft_consensus.cc:2468] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:32.789836 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.839758 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.609148 9596 raft_consensus.cc:1275] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260503 13:58:32.834718 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 349) took 2075 ms. Trace:
I20260503 13:58:32.829250 9450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 360) took 3551 ms. Trace:
I20260503 13:58:32.854288 9450 rpcz_store.cc:276] 0503 13:58:29.278041 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.762607 (+1484566us) service_pool.cc:224] Handling call
0503 13:58:32.373353 (+1610746us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:32.488810 (+115457us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.829055 (+340245us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b54004218d0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:32.829191 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3663317760}
I20260503 13:58:32.704769 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:32.760610 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.667109 10358 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.842519 9451 raft_consensus.cc:2410] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260503 13:58:32.806715 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.728446 9567 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:32.842867 9526 rpcz_store.cc:276] 0503 13:58:30.778135 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.851271 (+ 73136us) service_pool.cc:224] Handling call
0503 13:58:30.895600 (+ 44329us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:31.930865 (+1035265us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:31.930897 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:31.987708 (+ 56811us) log.cc:844] Serialized 26 byte log entry
0503 13:58:32.153762 (+166054us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:32.153837 (+ 75us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.822264 (+668427us) spinlock_profiling.cc:243] Waited 636 ms on lock 0x7b54003ff054. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:32.822314 (+ 50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:32.822361 (+ 47us) raft_consensus.cc:1685] finished
0503 13:58:32.822377 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:32.842140 (+ 19763us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":229031,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"spinlock_wait_cycles":1462768896,"thread_start_us":485861,"threads_started":2,"wal-append.queue_time_us":246001}
I20260503 13:58:32.843288 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.845175 9520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 427) took 2205 ms. Trace:
I20260503 13:58:32.858211 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.854089 9596 pending_rounds.cc:85] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868: Aborting all ops after (but not including) 1
I20260503 13:58:32.768307 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:32.789192 9491 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:32.846416 9528 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 6.
I20260503 13:58:32.692775 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.847368 9600 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:32.845134 10347 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:32.831893 9642 tablet_bootstrap.cc:492] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:32.806144 10569 consensus_queue.cc:1059] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260503 13:58:32.850613 9644 consensus_queue.cc:260] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.761281 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.873555 9527 raft_consensus.cc:2468] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:32.874399 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:32.874732 9444 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 418) took 2081 ms (client timeout 1551 ms). Trace:
W20260503 13:58:32.875120 9444 rpcz_store.cc:269] 0503 13:58:30.793002 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.812739 (+ 19737us) service_pool.cc:224] Handling call
0503 13:58:32.390386 (+1577647us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b5800000998. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50a3e 000000000035a3c9 00007f9f46a888e0 00007f9f46b51784 00007f9f46b517f1 00007f9f46aff760 00007f9f46aff044 00007f9f4006c834 00007f9f4006c7c2
0503 13:58:32.874360 (+483974us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b54003049a8. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f478f5d94 00007f9f46aff8e7 00007f9f46aff178 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:32.874663 (+ 303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":864080000}
I20260503 13:58:32.875404 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.851873 9519 raft_consensus.cc:2410] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260503 13:58:32.852285 9416 leader_election.cc:304] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:32.852608 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.812476 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.854192 9599 rpcz_store.cc:276] 0503 13:58:30.758839 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:32.470034 (+1711195us) service_pool.cc:224] Handling call
0503 13:58:32.470297 (+ 263us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:32.470598 (+ 301us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:32.470624 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:32.471034 (+ 410us) log.cc:844] Serialized 26 byte log entry
0503 13:58:32.610570 (+139536us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:32.610608 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.610669 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:32.834522 (+223853us) raft_consensus.cc:1685] finished
0503 13:58:32.834542 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:32.834652 (+ 110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":198044,"raft.queue_time_us":654,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":11201,"thread_start_us":132988,"threads_started":1,"wal-append.queue_time_us":226763}
I20260503 13:58:32.697983 10251 raft_consensus.cc:493] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:32.820181 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.855686 9598 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:32.702159 10561 raft_consensus.cc:2804] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:32.786921 10544 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.693188 10543 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.857910 10560 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:32.858067 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 129720us
I20260503 13:58:32.858146 9520 rpcz_store.cc:276] 0503 13:58:30.639518 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:30.639637 (+ 119us) service_pool.cc:224] Handling call
0503 13:58:30.639986 (+ 349us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:31.484297 (+844311us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:31.557189 (+ 72892us) log.cc:844] Serialized 14 byte log entry
0503 13:58:31.868280 (+311091us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:31.978934 (+110654us) log.cc:844] Serialized 26 byte log entry
0503 13:58:31.979067 (+ 133us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:31.979517 (+ 450us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.822272 (+842755us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54003fe154. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:32.822320 (+ 48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:32.822367 (+ 47us) raft_consensus.cc:1685] finished
0503 13:58:32.845012 (+ 22645us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:32.845127 (+ 115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4882,"prepare.queue_time_us":329822,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":221710,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":30818,"spinlock_wait_cycles":277297536,"thread_start_us":1003174,"threads_started":3,"wal-append.queue_time_us":649689}
I20260503 13:58:32.845439 9601 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:32.859128 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.840709 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.838306 9452 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:32.871582 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.871604 9642 ts_tablet_manager.cc:1403] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.736s user 0.022s sys 0.030s
I20260503 13:58:32.886400 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.892874 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.893429 9450 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:32.879961 10358 raft_consensus.cc:2804] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:32.871980 9453 raft_consensus.cc:3060] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:32.871212 10534 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.872942 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.878768 9527 raft_consensus.cc:2393] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:32.878611 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.897347 9599 raft_consensus.cc:2393] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:32.882701 10424 raft_consensus.cc:2804] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:32.855964 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.860272 9596 pending_rounds.cc:107] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260503 13:58:32.871865 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:32.899564 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:32.872478 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 83353us
I20260503 13:58:32.881732 9416 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:32.876260 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.874742 10347 raft_consensus.cc:3060] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:32.901965 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.902379 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.857789 10535 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.882745 9521 raft_consensus.cc:2410] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:32.845916 10243 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:32.885795 9644 ts_tablet_manager.cc:1434] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.898s user 0.006s sys 0.002s
I20260503 13:58:32.886308 9445 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
W20260503 13:58:32.905398 9644 ts_tablet_manager.cc:1467] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6: Tablet startup took 917ms
I20260503 13:58:32.860694 10396 raft_consensus.cc:2804] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:32.887526 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.887614 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.854537 10179 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.702308 10475 consensus_queue.cc:1048] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.157s
I20260503 13:58:32.857085 9522 raft_consensus.cc:2468] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:32.895923 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.917851 9520 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:32.884232 10251 raft_consensus.cc:515] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.899492 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.898038 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:32.900310 9601 raft_consensus.cc:2393] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:32.901607 9416 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:32.919682 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.904157 9491 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:32.923182 9453 raft_consensus.cc:2468] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:32.900095 9600 raft_consensus.cc:2376] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5. Re-sending same reply.
I20260503 13:58:32.906315 9519 raft_consensus.cc:2410] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:32.906631 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.908417 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.884853 10445 consensus_queue.cc:1048] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260503 13:58:32.904613 10537 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:32.894846 10358 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:32.915788 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.916035 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.938128 9491 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:32.881843 10561 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:32.900835 10424 raft_consensus.cc:493] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:32.899837 9596 raft_consensus.cc:2889] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:32.923964 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.925138 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:32.924705 10534 raft_consensus.cc:2749] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:32.954059 9521 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:32.940604 10358 raft_consensus.cc:3060] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:32.925202 9527 raft_consensus.cc:1275] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:32.907606 10243 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:32.908612 10396 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:32.914309 9418 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:32.894966 10544 consensus_queue.cc:1048] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260503 13:58:32.924575 10372 raft_consensus.cc:493] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:32.941160 9520 raft_consensus.cc:1240] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260503 13:58:32.918776 10535 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:32.941376 9524 raft_consensus.cc:2468] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:32.968019 10396 raft_consensus.cc:3060] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
W20260503 13:58:32.970902 10319 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260503 13:58:32.971230 10535 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.941804 10251 leader_election.cc:290] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.943539 10179 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.944209 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.973228 10535 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:32.945099 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.967986 9596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 335) took 4246 ms. Trace:
I20260503 13:58:32.947091 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.973902 9596 rpcz_store.cc:276] 0503 13:58:28.721636 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:29.594845 (+873209us) service_pool.cc:224] Handling call
0503 13:58:29.595130 (+ 285us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:31.922255 (+2327125us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b54000fab20. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41aa31a0 00007f9f41b2d5a0 00007f9f41b2e611 00007f9f41b2b16f 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c
0503 13:58:32.383757 (+461502us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54000fab20. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41aa944f 00007f9f41b22340 00007f9f41b23bbe 00007f9f41b2df29 00007f9f41b2e7f6 00007f9f41b2b16f 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2
0503 13:58:32.965123 (+581366us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:32.967932 (+ 2809us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":310881536}
I20260503 13:58:32.975064 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:32.938118 9599 raft_consensus.cc:2376] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4. Re-sending same reply.
I20260503 13:58:32.947960 9451 raft_consensus.cc:2410] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260503 13:58:32.905704 9644 ts_tablet_manager.cc:1469] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6: Trace:
0503 13:58:05.353369 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.221972 (+868603us) service_pool.cc:224] Handling call
0503 13:58:06.238810 (+ 16838us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.238897 (+ 87us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.360654 (+121757us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.399216 (+ 38562us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.399616 (+ 400us) inbound_call.cc:177] Queueing success response
0503 13:58:29.312601 (+22912985us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0503 13:58:31.987802 (+2675201us) ts_tablet_manager.cc:1435] Starting tablet replica
0503 13:58:32.839149 (+851347us) tablet_replica.cc:261] Starting instrumentation
0503 13:58:32.839624 (+ 475us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":705506,"raft.queue_time_us":4705,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":5989,"tablet-open.queue_time_us":22913028}
I20260503 13:58:32.975564 9596 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 4
W20260503 13:58:32.948542 9491 leader_election.cc:343] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:32.927708 9523 raft_consensus.cc:2393] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 6: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:32.927371 10506 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:32.985883 9521 raft_consensus.cc:2410] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:32.950166 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.941505 9351 catalog_manager.cc:5671] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 1 to 3, leader changed from 4bea375a55034750a1f8804b0db97868 (127.8.26.195) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 3 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:32.946638 9522 raft_consensus.cc:1275] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:32.928470 10511 raft_consensus.cc:2804] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:32.949039 10253 consensus_queue.cc:1048] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20260503 13:58:32.958218 9566 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:32.953190 9600 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:32.987664 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.989358 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.989666 9520 raft_consensus.cc:2410] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:32.952044 10424 raft_consensus.cc:3060] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:32.952725 10560 raft_consensus.cc:2804] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:32.944805 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:32.993510 9601 raft_consensus.cc:3060] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:32.949165 10561 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:32.925822 10580 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:32.978806 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 64578.6us
I20260503 13:58:32.986182 9643 raft_consensus.cc:359] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.985428 9642 raft_consensus.cc:359] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.995198 9642 raft_consensus.cc:385] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:32.995465 9642 raft_consensus.cc:740] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:32.885062 9448 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:32.964058 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:32.989734 9566 leader_election.cc:400] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599) with higher term. Message: Invalid argument: T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4.
I20260503 13:58:32.996196 9566 leader_election.cc:403] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260503 13:58:32.990113 9600 raft_consensus.cc:2468] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:32.996616 9418 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:32.978931 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:32.997869 9566 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:32.997296 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:32.984181 10587 raft_consensus.cc:2804] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:32.997764 9642 consensus_queue.cc:260] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:32.999454 9566 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:32.999879 9418 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:33.018111 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:33.029222 9600 raft_consensus.cc:3060] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:32.986968 10543 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:33.023615 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.029460 9566 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:32.987110 10511 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:33.030025 9418 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:32.978511 10372 raft_consensus.cc:515] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.030483 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.031663 9566 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.021533 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.006335 9527 raft_consensus.cc:2393] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:32.996068 9643 raft_consensus.cc:385] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:33.034966 9418 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:33.039853 9643 raft_consensus.cc:740] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:33.019969 9522 pending_rounds.cc:85] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a: Aborting all ops after (but not including) 2
W20260503 13:58:33.018070 10608 log.cc:927] Time spent T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260503 13:58:32.995620 10560 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:33.027864 10587 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:33.031435 9444 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:33.074105 10560 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:33.064698 10445 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:33.065789 9418 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:33.064416 9642 ts_tablet_manager.cc:1434] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.172s user 0.006s sys 0.006s
I20260503 13:58:33.067557 9643 consensus_queue.cc:260] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.068625 10475 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:33.082831 9593 raft_consensus.cc:2468] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:33.083093 9642 tablet_bootstrap.cc:492] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:33.070122 9566 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.072181 9601 raft_consensus.cc:2468] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:33.073377 9522 pending_rounds.cc:107] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260503 13:58:33.076730 9351 catalog_manager.cc:5671] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 0 to 2, leader changed from <none> to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 2 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:33.024739 10544 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:33.074150 10587 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:33.024857 10179 raft_consensus.cc:3060] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:33.062467 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.032753 10562 raft_consensus.cc:2804] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:33.033355 10319 raft_consensus.cc:2749] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:33.034267 10511 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:33.034106 10199 raft_consensus.cc:2804] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:33.040517 10486 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:33.079231 9418 leader_election.cc:304] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:33.087548 9449 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:32.994763 10506 raft_consensus.cc:2804] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:33.088935 9522 raft_consensus.cc:2889] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:33.022128 10396 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.091408 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:33.091990 9492 leader_election.cc:336] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Timed out: RequestConsensusVote RPC to 127.8.26.193:45599 timed out after 2.343s (SENT)
I20260503 13:58:33.095454 10562 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:33.097787 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.077273 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.100886 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.084293 10372 leader_election.cc:290] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:33.082872 10475 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:33.077782 10445 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
W20260503 13:58:33.129199 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:33.105707 10486 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:33.109160 9600 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:33.089175 10300 raft_consensus.cc:2804] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:33.111426 9519 raft_consensus.cc:1240] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:32.996806 10547 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.105710 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.106978 9566 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.090708 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.106501 9351 catalog_manager.cc:5671] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 428b342dc4f041df956015e7d7cd173a (127.8.26.194). New cstate: current_term: 1 leader_uuid: "428b342dc4f041df956015e7d7cd173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } }
I20260503 13:58:33.113703 10542 raft_consensus.cc:2804] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:33.088406 10544 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:33.124243 9643 ts_tablet_manager.cc:1434] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.450s user 0.012s sys 0.021s
I20260503 13:58:33.124429 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.093204 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.105055 10558 raft_consensus.cc:3055] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:33.128293 9451 raft_consensus.cc:2393] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 5: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:33.106243 9418 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:33.164625 9351 catalog_manager.cc:5671] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 4 to 5, leader changed from 428b342dc4f041df956015e7d7cd173a (127.8.26.194) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 5 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
W20260503 13:58:33.168879 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 62718.8us
I20260503 13:58:33.169557 9418 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:33.110337 10506 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:33.129843 9596 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:33.115087 10564 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:33.158645 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:33.159199 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 52289.4us
I20260503 13:58:33.176084 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:33.176548 9566 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.159443 10253 raft_consensus.cc:2749] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:33.161963 9642 tablet_bootstrap.cc:654] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:33.166329 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.126911 10562 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:33.172602 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:33.120901 10319 consensus_queue.cc:1048] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260503 13:58:33.169708 9351 catalog_manager.cc:5671] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 1 to 2, leader changed from 4bea375a55034750a1f8804b0db97868 (127.8.26.195) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 2 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:33.185041 9351 catalog_manager.cc:5671] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 1 to 3, leader changed from 428b342dc4f041df956015e7d7cd173a (127.8.26.194) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 3 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:33.113801 10199 raft_consensus.cc:697] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:33.174602 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.174696 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.187403 9451 raft_consensus.cc:2468] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:33.178149 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.178745 9445 raft_consensus.cc:3060] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:33.131829 10475 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:33.182476 9525 raft_consensus.cc:2393] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:33.188856 9600 raft_consensus.cc:2468] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:33.200734 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.201409 9600 raft_consensus.cc:2368] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4.
I20260503 13:58:33.164888 10445 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:33.164304 10486 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:33.127009 10463 consensus_queue.cc:1048] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260503 13:58:33.219468 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.214927 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.206157 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:33.220126 9600 raft_consensus.cc:2468] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:33.215158 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.220971 9643 tablet_bootstrap.cc:492] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:33.221019 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.167766 10544 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:33.167608 10558 raft_consensus.cc:740] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:33.177367 10474 raft_consensus.cc:2749] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599) with higher term. Message: Invalid argument: T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4.
I20260503 13:58:33.179090 10372 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260503 13:58:33.171272 10295 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:33.167702 10300 raft_consensus.cc:697] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:33.173278 10506 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:33.209972 10283 raft_consensus.cc:2804] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:33.223150 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.225637 9567 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.171276 9448 raft_consensus.cc:2468] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:33.215142 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.226591 9644 tablet_bootstrap.cc:492] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:33.232709 9491 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:33.233328 9416 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.234287 10558 consensus_queue.cc:260] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.236943 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:33.226928 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:33.216949 10609 log.cc:927] Time spent T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260503 13:58:33.259128 9491 leader_election.cc:304] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:33.244328 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:33.235761 10372 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:33.226686 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.240658 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.220109 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.218218 10580 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:33.257911 10463 raft_consensus.cc:3060] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:33.240712 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.270288 9445 raft_consensus.cc:2468] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:33.308333 9449 raft_consensus.cc:2468] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:33.239560 10283 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:33.293061 9491 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
W20260503 13:58:33.301898 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:33.261488 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.309226 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:33.308703 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 132215us
I20260503 13:58:33.308720 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.317992 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 425) took 1144 ms. Trace:
I20260503 13:58:33.268354 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.245319 10385 raft_consensus.cc:2804] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:33.265422 10372 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:33.329255 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:33.239950 10295 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:33.332741 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 162112us
I20260503 13:58:33.315124 10283 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:33.318303 9445 rpcz_store.cc:276] 0503 13:58:32.173843 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.308465 (+1134622us) service_pool.cc:224] Handling call
0503 13:58:33.317937 (+ 9472us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:33.340432 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.340605 9491 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:33.310906 10558 raft_consensus.cc:3060] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:33.341030 9445 raft_consensus.cc:2468] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:33.305688 10543 raft_consensus.cc:2749] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260503 13:58:33.295212 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.320541 9566 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.311200 10535 raft_consensus.cc:2804] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:33.331926 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.351681 9449 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:33.352547 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.353191 9449 raft_consensus.cc:2393] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:33.353878 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.354357 9449 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:33.325305 10396 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:33.313822 10170 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:33.186805 9351 catalog_manager.cc:5671] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 1 to 2, leader changed from 428b342dc4f041df956015e7d7cd173a (127.8.26.194) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 2 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:33.350075 9566 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.319365 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.357683 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.377539 9449 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:33.378374 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.378906 9449 raft_consensus.cc:2468] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:33.379601 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.379354 9351 catalog_manager.cc:5671] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 2 to 4, leader changed from 428b342dc4f041df956015e7d7cd173a (127.8.26.194) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 4 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:33.293295 10199 consensus_queue.cc:237] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.330713 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.343974 9491 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
W20260503 13:58:33.389415 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 45492.7us
I20260503 13:58:33.390086 9491 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:33.362528 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.385265 9492 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:33.391137 9445 raft_consensus.cc:2468] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:33.363958 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.373344 9597 raft_consensus.cc:2393] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:33.330538 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 433) took 1143 ms. Trace:
I20260503 13:58:33.380903 9351 catalog_manager.cc:5671] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 3 to 4, leader changed from 428b342dc4f041df956015e7d7cd173a (127.8.26.194) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 4 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:33.351475 10569 raft_consensus.cc:2804] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:33.349059 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:33.401181 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 167915us
I20260503 13:58:33.393967 9566 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.403661 9566 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.355098 10580 raft_consensus.cc:2749] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:33.397524 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:33.377427 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:33.405491 9445 raft_consensus.cc:3060] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:33.380515 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:33.312747 10300 consensus_queue.cc:237] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.406054 9449 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:33.407478 9449 raft_consensus.cc:1240] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
W20260503 13:58:33.400066 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 230576us
I20260503 13:58:33.411013 9418 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:33.398841 9351 catalog_manager.cc:5671] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 1 to 2, leader changed from 4bea375a55034750a1f8804b0db97868 (127.8.26.195) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 2 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:33.420395 9566 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.388541 10543 raft_consensus.cc:493] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:33.422794 9449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 422) took 1246 ms. Trace:
I20260503 13:58:33.423175 9449 rpcz_store.cc:276] 0503 13:58:32.167002 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.406865 (+1239863us) service_pool.cc:224] Handling call
0503 13:58:33.407244 (+ 379us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.412504 (+ 5260us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.412960 (+ 456us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:33.425678 9449 raft_consensus.cc:1275] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:33.426131 9566 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.428253 9351 catalog_manager.cc:5671] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 1 to 2, leader changed from 4bea375a55034750a1f8804b0db97868 (127.8.26.195) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 2 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:33.401719 10569 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:33.398954 10547 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:33.432933 9449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 424) took 1257 ms. Trace:
I20260503 13:58:33.379026 10562 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.422880 9596 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:33.420718 9447 raft_consensus.cc:2410] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 4 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260503 13:58:33.399013 10616 raft_consensus.cc:2749] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:33.447520 10616 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:33.447949 10616 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.434423 9491 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:33.386392 10170 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.435997 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.406170 9595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 343) took 2653 ms. Trace:
I20260503 13:58:33.413143 10396 raft_consensus.cc:2749] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:33.438292 9449 rpcz_store.cc:276] 0503 13:58:32.169433 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.424003 (+1254570us) service_pool.cc:224] Handling call
0503 13:58:33.424267 (+ 264us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.426119 (+ 1852us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.426294 (+ 175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:33.439029 9448 raft_consensus.cc:2410] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260503 13:58:33.433198 10319 raft_consensus.cc:2749] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:33.459004 9445 raft_consensus.cc:2468] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
W20260503 13:58:33.440972 10612 log.cc:927] Time spent T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.107s user 0.000s sys 0.001s
I20260503 13:58:33.397672 9446 rpcz_store.cc:276] 0503 13:58:32.187254 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.174727 (+987473us) service_pool.cc:224] Handling call
0503 13:58:33.324902 (+150175us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:33.444940 10613 log.cc:927] Time spent T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20260503 13:58:33.425155 10474 raft_consensus.cc:2749] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:33.433107 10580 raft_consensus.cc:2804] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:33.429725 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.464119 9596 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:33.480630 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:33.444377 10611 log.cc:927] Time spent T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260503 13:58:33.466046 9453 raft_consensus.cc:2393] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
W20260503 13:58:33.443696 10347 consensus_meta.cc:287] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.531s user 0.009s sys 0.001s
I20260503 13:58:33.446740 9351 catalog_manager.cc:5671] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 2 to 3, leader changed from 4bea375a55034750a1f8804b0db97868 (127.8.26.195) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 3 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:33.440322 10569 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:33.439724 10385 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:33.481494 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.427078 10543 raft_consensus.cc:515] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.452294 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.466602 10542 raft_consensus.cc:697] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:33.489428 9351 catalog_manager.cc:5671] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } }
I20260503 13:58:33.498278 9644 tablet_bootstrap.cc:654] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:33.498739 9351 catalog_manager.cc:5671] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 2 to 3, leader changed from 4bea375a55034750a1f8804b0db97868 (127.8.26.195) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 3 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:33.481259 9597 raft_consensus.cc:2468] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:33.455068 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 436) took 1243 ms. Trace:
I20260503 13:58:33.443895 10561 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.484701 10580 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:33.504344 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:33.508697 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 123496us
I20260503 13:58:33.504678 9351 catalog_manager.cc:5671] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 1 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:33.489813 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 377) took 1234 ms. Trace:
I20260503 13:58:33.493435 10385 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:33.512799 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.465843 10475 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.512109 9492 leader_election.cc:304] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:33.515793 9597 raft_consensus.cc:2393] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:33.512100 9351 catalog_manager.cc:5671] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 reported cstate change: term changed from 1 to 2, leader changed from 4bea375a55034750a1f8804b0db97868 (127.8.26.195) to 1cb63886040047559d3893923cc9aaf6 (127.8.26.193). New cstate: current_term: 2 leader_uuid: "1cb63886040047559d3893923cc9aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260503 13:58:33.550191 9521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 455) took 1168 ms. Trace:
I20260503 13:58:33.551190 9521 rpcz_store.cc:276] 0503 13:58:32.381506 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:32.990700 (+609194us) service_pool.cc:224] Handling call
0503 13:58:32.990963 (+ 263us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:32.991223 (+ 260us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:32.991239 (+ 16us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:32.991551 (+ 312us) log.cc:844] Serialized 26 byte log entry
0503 13:58:33.240805 (+249254us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:33.240842 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.241027 (+ 185us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:33.549977 (+308950us) raft_consensus.cc:1685] finished
0503 13:58:33.549998 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:33.550130 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":70,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":2130,"thread_start_us":220414,"threads_started":1,"wal-append.queue_time_us":309873}
I20260503 13:58:33.552281 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:33.555310 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 375) took 1305 ms. Trace:
I20260503 13:58:33.517661 9602 raft_consensus.cc:2393] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:33.556157 9492 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:33.555656 9453 rpcz_store.cc:276] 0503 13:58:32.250289 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.129703 (+879414us) service_pool.cc:224] Handling call
0503 13:58:33.555144 (+425441us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x7b540029f254. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:33.555260 (+ 116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":653189376}
I20260503 13:58:33.555974 9445 raft_consensus.cc:1240] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
PC: @ 0x0 (unknown)
I20260503 13:58:33.513283 10547 raft_consensus.cc:2804] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:33.491602 10347 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.511108 10295 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:33.517138 10336 consensus_meta.cc:287] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.668s user 0.006s sys 0.006s
W20260503 13:58:33.557513 10614 log.cc:927] Time spent T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.213s user 0.000s sys 0.001s
I20260503 13:58:33.563095 9492 leader_election.cc:304] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:33.562565 9519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 394) took 1149 ms. Trace:
I20260503 13:58:33.502231 10625 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:33.510447 10580 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
W20260503 13:58:33.564742 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 153784us
W20260503 13:58:33.562112 10424 consensus_meta.cc:287] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.563s user 0.004s sys 0.004s
I20260503 13:58:33.566298 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:33.568679 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:33.567641 9519 rpcz_store.cc:276] 0503 13:58:32.412981 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.165560 (+752579us) service_pool.cc:224] Handling call
0503 13:58:33.180081 (+ 14521us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.180280 (+ 199us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:33.342522 (+162242us) log.cc:844] Serialized 14 byte log entry
0503 13:58:33.562263 (+219741us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:33.562309 (+ 46us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:33.562347 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.562408 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:33.562511 (+ 103us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13419,"prepare.queue_time_us":178586,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":113627,"raft.run_cpu_time_us":68,"raft.run_wall_time_us":3438,"thread_start_us":344457,"threads_started":3,"wal-append.queue_time_us":79034}
I20260503 13:58:33.574427 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:33.584547 9453 raft_consensus.cc:3060] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:33.564702 10534 raft_consensus.cc:2804] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:33.574193 10631 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:33.574616 9418 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:33.563652 9523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 447) took 1236 ms. Trace:
I20260503 13:58:33.501873 9447 rpcz_store.cc:276] 0503 13:58:32.208777 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:32.973262 (+764485us) service_pool.cc:224] Handling call
0503 13:58:33.263843 (+290581us) spinlock_profiling.cc:243] Waited 5.65 ms on lock 0x7b5800000998. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46aff760 00007f9f46aff044 00007f9f4006c834 00007f9f4006c7c2
0503 13:58:33.452485 (+188642us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12990208}
I20260503 13:58:33.590288 9527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 379) took 1181 ms. Trace:
I20260503 13:58:33.560478 10632 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:33.569165 10361 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:33.569054 10547 raft_consensus.cc:697] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
W20260503 13:58:33.591704 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 158775us
I20260503 13:58:33.591681 9527 rpcz_store.cc:276] 0503 13:58:32.409092 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.065340 (+656248us) service_pool.cc:224] Handling call
0503 13:58:33.065646 (+ 306us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:33.065936 (+ 290us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:33.065962 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:33.066381 (+ 419us) log.cc:844] Serialized 26 byte log entry
0503 13:58:33.220181 (+153800us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:33.220217 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.220285 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:33.590076 (+369791us) raft_consensus.cc:1685] finished
0503 13:58:33.590097 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:33.590214 (+ 117us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1110,"raft.run_cpu_time_us":50,"raft.run_wall_time_us":50,"spinlock_wait_cycles":1970560,"thread_start_us":153431,"threads_started":1,"wal-append.queue_time_us":246159}
I20260503 13:58:33.566854 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.602505 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 393) took 1190 ms. Trace:
I20260503 13:58:33.567080 10633 raft_consensus.cc:3060] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:33.572826 10625 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:33.593698 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:33.601667 9453 raft_consensus.cc:2468] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:33.591499 10631 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.576285 10542 consensus_queue.cc:237] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:33.633983 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:33.608843 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 404) took 1100 ms. Trace:
W20260503 13:58:33.636600 10537 consensus_meta.cc:287] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.697s user 0.006s sys 0.013s
I20260503 13:58:33.640112 10635 raft_consensus.cc:2804] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:33.640429 10537 raft_consensus.cc:2749] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:33.513700 9448 rpcz_store.cc:276] 0503 13:58:32.250756 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.264321 (+1013565us) service_pool.cc:224] Handling call
0503 13:58:33.363863 (+ 99542us) spinlock_profiling.cc:243] Waited 6.35 ms on lock 0x7b5800000998. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50a3e 000000000035a3c9 00007f9f46a888e0 00007f9f46b51784 00007f9f46b517f1 00007f9f46aff760 00007f9f46aff044 00007f9f4006c834 00007f9f4006c7c2
0503 13:58:33.485356 (+121493us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14601472}
I20260503 13:58:33.603241 9491 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:33.639129 9525 rpcz_store.cc:276] 0503 13:58:32.508389 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.258576 (+750187us) service_pool.cc:224] Handling call
0503 13:58:33.258858 (+ 282us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:33.291390 (+ 32532us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:33.291421 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:33.292597 (+ 1176us) log.cc:844] Serialized 26 byte log entry
0503 13:58:33.404754 (+112157us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:33.404797 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.404866 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:33.608652 (+203786us) raft_consensus.cc:1685] finished
0503 13:58:33.608669 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:33.608792 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23341,"raft.queue_time_us":45464,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":2904,"thread_start_us":53855,"threads_started":1,"wal-append.queue_time_us":194021}
I20260503 13:58:33.602870 9528 rpcz_store.cc:276] 0503 13:58:32.412702 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.123888 (+711186us) service_pool.cc:224] Handling call
0503 13:58:33.157475 (+ 33587us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.157663 (+ 188us) raft_consensus.cc:1534] Early marking committed up to index 4
0503 13:58:33.308173 (+150510us) log.cc:844] Serialized 14 byte log entry
0503 13:58:33.602216 (+294043us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:33.602262 (+ 46us) raft_consensus.cc:1645] Marking committed up to 4
0503 13:58:33.602299 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.602355 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:33.602453 (+ 98us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13028,"prepare.queue_time_us":176853,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":2839,"thread_start_us":409179,"threads_started":2}
W20260503 13:58:33.589063 10459 consensus_meta.cc:287] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.740s user 0.007s sys 0.003s
I20260503 13:58:33.578397 10424 raft_consensus.cc:515] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.649108 9643 tablet_bootstrap.cc:654] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:33.668823 10535 raft_consensus.cc:697] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:33.592870 10629 raft_consensus.cc:2804] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:33.594781 10543 leader_election.cc:290] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:33.622390 10638 raft_consensus.cc:3060] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:33.640774 9594 raft_consensus.cc:1240] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: []
W20260503 13:58:33.649600 10511 consensus_meta.cc:287] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.554s user 0.009s sys 0.007s
I20260503 13:58:33.637794 10562 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:33.595240 10632 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:33.600850 10618 consensus_queue.cc:1059] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.066s
I20260503 13:58:33.640547 10635 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:33.623759 9593 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:33.695674 9602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 389) took 1003 ms. Trace:
I20260503 13:58:33.702082 9602 rpcz_store.cc:276] 0503 13:58:32.692450 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:32.699515 (+ 7065us) service_pool.cc:224] Handling call
0503 13:58:33.695447 (+995932us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b5400077e54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:33.695610 (+ 163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1517162368}
I20260503 13:58:33.699438 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 464) took 1027 ms. Trace:
W20260503 13:58:33.703503 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 128938us
I20260503 13:58:33.637720 10625 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
W20260503 13:58:33.701223 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 138155us
I20260503 13:58:33.703135 9528 rpcz_store.cc:276] 0503 13:58:32.672229 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.665970 (+993741us) service_pool.cc:224] Handling call
0503 13:58:33.698986 (+ 33016us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.699138 (+ 152us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:33.699162 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:33.699198 (+ 36us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:33.699225 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.699274 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:33.699388 (+ 114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:33.704991 9492 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:33.605487 10361 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.673018 10170 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:33.708496 9418 leader_election.cc:304] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:33.672822 10636 raft_consensus.cc:2749] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:33.454226 9595 rpcz_store.cc:276] 0503 13:58:30.752873 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:32.435197 (+1682324us) service_pool.cc:224] Handling call
0503 13:58:32.435510 (+ 313us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:32.781156 (+345646us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:32.786888 (+ 5732us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:32.844917 (+ 58029us) log.cc:844] Serialized 26 byte log entry
0503 13:58:32.991780 (+146863us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:32.995287 (+ 3507us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.022914 (+ 27627us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:33.390412 (+367498us) raft_consensus.cc:1685] finished
0503 13:58:33.392238 (+ 1826us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:33.401883 (+ 9645us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4027,"raft.queue_time_us":66612,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":11632,"thread_start_us":102698,"threads_started":1,"wal-append.queue_time_us":253991}
I20260503 13:58:33.700490 10632 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:33.701457 10537 raft_consensus.cc:2804] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:33.730865 9526 raft_consensus.cc:2410] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260503 13:58:33.692902 10629 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:33.715286 9492 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:33.716368 10639 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:33.710444 9444 raft_consensus.cc:2368] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2.
I20260503 13:58:33.701414 10635 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
W20260503 13:58:33.728531 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 125362us
I20260503 13:58:33.672840 10459 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
W20260503 13:58:33.717731 10623 log.cc:927] Time spent T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.233s user 0.001s sys 0.000s
I20260503 13:58:33.710603 10562 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:33.698743 10511 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.739019 9491 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:33.740269 9492 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:33.718613 10561 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:33.719726 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 358) took 2953 ms. Trace:
I20260503 13:58:33.726459 10170 raft_consensus.cc:3055] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:33.714053 10644 raft_consensus.cc:2749] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:33.725068 10634 raft_consensus.cc:2804] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election won for term 4
W20260503 13:58:33.771577 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 363231us
I20260503 13:58:33.772701 9416 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:33.734794 10537 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:33.736030 10475 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:33.738000 10629 raft_consensus.cc:3060] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:33.739711 10620 raft_consensus.cc:2804] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:33.772794 10562 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
*** SIGABRT (@0x3e80000206b) received by PID 8299 (TID 0x7f9f3a67b840) from PID 8299; stack trace: ***
W20260503 13:58:33.796515 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 57546.5us
W20260503 13:58:33.797312 9491 leader_election.cc:343] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:33.767843 10424 leader_election.cc:290] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:33.785952 10459 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
W20260503 13:58:33.788632 10358 consensus_meta.cc:287] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.806s user 0.007s sys 0.007s
I20260503 13:58:33.773784 10347 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:33.790508 10587 consensus_meta.cc:287] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.700s user 0.001s sys 0.013s
W20260503 13:58:33.793044 10637 log.cc:927] Time spent T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260503 13:58:33.792958 9444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 451) took 1356 ms. Trace:
@ 0x2cd620 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
W20260503 13:58:33.793361 10445 consensus_meta.cc:287] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.573s user 0.009s sys 0.000s
I20260503 13:58:33.731933 10547 consensus_queue.cc:237] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:33.768122 10243 consensus_meta.cc:287] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.791s user 0.004s sys 0.004s
I20260503 13:58:33.795521 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 443) took 1533 ms. Trace:
@ 0x2cfb03 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20260503 13:58:33.795668 10631 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:33.776873 9599 raft_consensus.cc:2393] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:33.796514 9453 raft_consensus.cc:1240] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:33.741864 10616 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:33.749323 10336 raft_consensus.cc:2749] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260503 13:58:33.798432 9491 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:33.798368 9451 raft_consensus.cc:2393] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:33.782871 9566 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:33.772717 10610 log.cc:927] Time spent T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.431s user 0.003s sys 0.000s
I20260503 13:58:33.742161 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 374) took 1490 ms. Trace:
I20260503 13:58:33.769937 10170 raft_consensus.cc:740] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:33.777094 10537 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:33.799311 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 351) took 3035 ms. Trace:
W20260503 13:58:33.801487 9416 leader_election.cc:343] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:33.805781 9522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 459) took 1413 ms. Trace:
W20260503 13:58:33.804212 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 79219.4us
I20260503 13:58:33.738898 10639 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.803689 10644 raft_consensus.cc:2804] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:33.812816 9566 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.814524 9416 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.801954 9593 raft_consensus.cc:1275] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260503 13:58:33.813951 9597 rpcz_store.cc:276] 0503 13:58:30.764830 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.562269 (+2797439us) service_pool.cc:224] Handling call
0503 13:58:33.607753 (+ 45484us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x7b5800010898. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:33.607919 (+ 166us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.608083 (+ 164us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:33.608350 (+ 267us) log.cc:844] Serialized 14 byte log entry
0503 13:58:33.798998 (+190648us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:33.799048 (+ 50us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:33.799089 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.799151 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:33.799257 (+ 106us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":142,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":71,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":1129,"spinlock_wait_cycles":11416704,"thread_start_us":190339,"threads_started":1}
I20260503 13:58:33.826092 9566 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.826897 9597 raft_consensus.cc:1240] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:33.827411 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 341) took 3033 ms. Trace:
I20260503 13:58:33.827675 9597 rpcz_store.cc:276] 0503 13:58:30.794416 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.826465 (+3032049us) service_pool.cc:224] Handling call
0503 13:58:33.826738 (+ 273us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.827211 (+ 473us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.827366 (+ 155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:33.810264 10634 raft_consensus.cc:697] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:33.802060 10300 raft_consensus.cc:2749] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:33.798739 10620 raft_consensus.cc:493] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:33.743338 10535 consensus_queue.cc:237] T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.824182 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:33.807919 10243 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.805543 10587 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:33.815022 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 74814.5us
I20260503 13:58:33.842525 9566 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.769810 9594 rpcz_store.cc:276] 0503 13:58:30.766073 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.573144 (+2807071us) service_pool.cc:224] Handling call
0503 13:58:33.593935 (+ 20791us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.701885 (+107950us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.715317 (+ 13432us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:33.847147 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 48777.3us
I20260503 13:58:33.848218 10646 raft_consensus.cc:2749] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:33.806957 10445 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.802959 10358 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.800730 10580 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.824885 10459 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:33.828730 9597 raft_consensus.cc:1240] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:33.853977 9520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 457) took 1471 ms. Trace:
I20260503 13:58:33.856441 9416 leader_election.cc:304] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:33.884115 9642 tablet_bootstrap.cc:492] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:33.884591 9642 ts_tablet_manager.cc:1403] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.802s user 0.019s sys 0.005s
I20260503 13:58:33.825281 10616 consensus_queue.cc:1059] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.034s
I20260503 13:58:33.825352 10543 raft_consensus.cc:2804] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:33.848392 10644 raft_consensus.cc:697] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:33.858717 9491 leader_election.cc:304] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:33.845296 10253 raft_consensus.cc:2749] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:33.842221 10561 raft_consensus.cc:2749] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260503 13:58:33.859433 9492 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:33.872223 9566 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:33.873603 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:33.853339 10179 consensus_meta.cc:287] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.759s user 0.008s sys 0.001s
I20260503 13:58:33.806643 9448 rpcz_store.cc:276] 0503 13:58:32.262465 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.674560 (+1412095us) service_pool.cc:224] Handling call
0503 13:58:33.728829 (+ 54269us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.775026 (+ 46197us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.795446 (+ 20420us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:33.875020 9449 raft_consensus.cc:1240] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:33.875169 10193 raft_consensus.cc:2749] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:33.803126 9444 rpcz_store.cc:276] 0503 13:58:32.436391 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.310975 (+874584us) service_pool.cc:224] Handling call
0503 13:58:33.792757 (+481782us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b5400294ad4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:33.792901 (+ 144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":710936832}
I20260503 13:58:33.850193 10636 raft_consensus.cc:2749] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260503 13:58:33.825263 10475 consensus_queue.cc:1048] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260503 13:58:33.882789 9598 raft_consensus.cc:2368] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 5. Current term is 6.
I20260503 13:58:33.901988 9596 raft_consensus.cc:2393] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:33.853852 10620 raft_consensus.cc:3060] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:33.882859 9520 rpcz_store.cc:276] 0503 13:58:32.382286 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:32.992888 (+610602us) service_pool.cc:224] Handling call
0503 13:58:33.018438 (+ 25550us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:33.018756 (+ 318us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:33.018782 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:33.019157 (+ 375us) log.cc:844] Serialized 26 byte log entry
0503 13:58:33.206782 (+187625us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:33.366069 (+159287us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b1800304120. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f416d285c 00007f9f3ea70c0b 00007f9f47869bc9 00007f9f3ea8ccdb 00007f9f3ea8af7f 00007f9f3ea8aa95 00007f9f4552e837 00007f9f41b35d87 00007f9f41b4549f
0503 13:58:33.366192 (+ 123us) log.cc:844] Serialized 14 byte log entry
0503 13:58:33.558672 (+192480us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.558757 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:33.604773 (+ 46016us) spinlock_profiling.cc:243] Waited 5.91 ms on lock 0x7b1800304120. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea8da33 00007f9f3ea8fe01 00007f9f3ea8fdb9 00007f9f3ea8fd49 00007f9f3ea8fd11 00007f9f3ea8f00d 0000000000379254 0000000000379099 00007f9f3ea70fba
0503 13:58:33.772220 (+167447us) spinlock_profiling.cc:243] Waited 6.45 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:33.853764 (+ 81544us) raft_consensus.cc:1685] finished
0503 13:58:33.853786 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:33.853924 (+ 138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8386,"prepare.queue_time_us":179526,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":87557,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":19053,"spinlock_wait_cycles":31905792,"thread_start_us":467744,"threads_started":3,"wal-append.queue_time_us":287286}
I20260503 13:58:33.889999 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 342) took 3076 ms. Trace:
I20260503 13:58:33.891129 9491 leader_election.cc:304] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
W20260503 13:58:33.908138 9491 leader_election.cc:343] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:33.908746 9491 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:33.909650 9491 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:33.911648 10652 consensus_queue.cc:1048] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:33.896251 9492 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:33.912725 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:33.892503 9644 tablet_bootstrap.cc:492] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
W20260503 13:58:33.912982 10642 log.cc:927] Time spent T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20260503 13:58:33.913426 9644 ts_tablet_manager.cc:1403] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.687s user 0.016s sys 0.014s
I20260503 13:58:33.914801 10639 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:33.915947 9644 raft_consensus.cc:359] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.916785 10655 raft_consensus.cc:2764] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260503 13:58:33.917397 10655 raft_consensus.cc:2804] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election won for term 6
I20260503 13:58:33.917840 10655 raft_consensus.cc:697] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 6 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:33.918619 10655 consensus_queue.cc:237] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:33.919263 10283 consensus_meta.cc:287] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.568s user 0.003s sys 0.005s
I20260503 13:58:33.919624 10283 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.920650 10511 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:33.856583 9593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 337) took 3090 ms. Trace:
I20260503 13:58:33.893747 10179 raft_consensus.cc:2749] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260503 13:58:33.882349 10372 consensus_meta.cc:287] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.542s user 0.007s sys 0.008s
I20260503 13:58:33.893337 10616 raft_consensus.cc:3055] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:33.882615 10459 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:33.893640 10649 raft_consensus.cc:2804] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:33.898399 10253 raft_consensus.cc:2804] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:33.898927 10445 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:33.853070 10641 consensus_queue.cc:1059] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 0, Time since last communication: 0.050s
I20260503 13:58:33.876444 10300 raft_consensus.cc:2764] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260503 13:58:33.883483 10486 consensus_meta.cc:287] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.643s user 0.006s sys 0.003s
I20260503 13:58:33.899638 10347 consensus_queue.cc:1048] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:33.589643 9523 rpcz_store.cc:276] 0503 13:58:32.324898 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:32.981427 (+656529us) service_pool.cc:224] Handling call
0503 13:58:32.981741 (+ 314us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:32.982031 (+ 290us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:32.982057 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:32.982467 (+ 410us) log.cc:844] Serialized 26 byte log entry
0503 13:58:33.236080 (+253613us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:33.334452 (+ 98372us) log.cc:844] Serialized 14 byte log entry
0503 13:58:33.339227 (+ 4775us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.339299 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:33.518481 (+179182us) raft_consensus.cc:1685] finished
0503 13:58:33.518502 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:33.561219 (+ 42717us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":167074,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":2576,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":4701,"thread_start_us":300132,"threads_started":2,"wal-append.queue_time_us":321646}
W20260503 13:58:33.935915 10564 consensus_meta.cc:287] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.753s user 0.005s sys 0.005s
I20260503 13:58:33.905056 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37228 (request call id 407) took 1395 ms. Trace:
I20260503 13:58:33.949084 9526 rpcz_store.cc:276] 0503 13:58:32.509264 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:32.896676 (+387412us) service_pool.cc:224] Handling call
0503 13:58:33.903230 (+1006554us) spinlock_profiling.cc:243] Waited 828 ms on lock 0x7b540004db54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:33.904997 (+ 1767us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1903783168}
I20260503 13:58:33.821107 9522 rpcz_store.cc:276] 0503 13:58:32.392066 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:32.998771 (+606705us) service_pool.cc:224] Handling call
0503 13:58:33.019804 (+ 21033us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:33.128336 (+108532us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:33.128396 (+ 60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:33.206636 (+ 78240us) log.cc:844] Serialized 26 byte log entry
0503 13:58:33.338918 (+132282us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:33.345154 (+ 6236us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.356035 (+ 10881us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:33.796157 (+440122us) raft_consensus.cc:1685] finished
0503 13:58:33.797945 (+ 1788us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:33.804336 (+ 6391us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":89822,"threads_started":1,"wal-append.queue_time_us":214430}
I20260503 13:58:33.896523 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:33.962074 9528 raft_consensus.cc:3060] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:33.924419 9593 rpcz_store.cc:276] 0503 13:58:30.766525 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.727040 (+2960515us) service_pool.cc:224] Handling call
0503 13:58:33.742370 (+ 15330us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.844596 (+102226us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.853815 (+ 9219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:33.889540 10543 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:33.923507 9492 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:33.897266 9524 raft_consensus.cc:1240] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:33.925531 9597 rpcz_store.cc:276] 0503 13:58:30.813665 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.828321 (+3014656us) service_pool.cc:224] Handling call
0503 13:58:33.828576 (+ 255us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.889734 (+ 61158us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.889941 (+ 207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:33.965355 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:33.926417 9644 raft_consensus.cc:385] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:33.966142 9644 raft_consensus.cc:740] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:33.924242 10358 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:33.900851 9452 raft_consensus.cc:1240] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:33.927350 9520 raft_consensus.cc:2468] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:33.895936 10653 raft_consensus.cc:2749] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:33.935225 10649 raft_consensus.cc:493] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:33.899538 10644 consensus_queue.cc:237] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.901401 10646 raft_consensus.cc:2749] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:33.900511 10587 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:33.855216 10170 consensus_queue.cc:260] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:33.934273 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 76658.8us
W20260503 13:58:33.927798 10560 consensus_meta.cc:287] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.840s user 0.013s sys 0.008s
W20260503 13:58:33.928364 10463 consensus_meta.cc:287] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.617s user 0.009s sys 0.000s
I20260503 13:58:33.942061 10562 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:33.924998 10336 raft_consensus.cc:2749] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:33.894994 10635 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.904074 9643 tablet_bootstrap.cc:492] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:33.952098 9642 raft_consensus.cc:359] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.996008 9642 raft_consensus.cc:385] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:33.996304 9642 raft_consensus.cc:740] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:33.931037 10616 raft_consensus.cc:740] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:33.971822 9644 consensus_queue.cc:260] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.972777 9597 raft_consensus.cc:2393] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:33.961410 10641 raft_consensus.cc:3055] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:33.976375 9567 leader_election.cc:304] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:33.973773 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 463) took 1301 ms. Trace:
I20260503 13:58:33.811345 9447 rpcz_store.cc:276] 0503 13:58:32.250052 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.635917 (+1385865us) service_pool.cc:224] Handling call
0503 13:58:33.675659 (+ 39742us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.709554 (+ 33895us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:33.715002 (+ 5448us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:33.719340 (+ 4338us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:33.728536 (+ 9196us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.735098 (+ 6562us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:33.740630 (+ 5532us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":727,"raft.queue_time_us":29090,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":3489}
I20260503 13:58:33.989014 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:33.940130 10253 raft_consensus.cc:697] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:33.933140 10372 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.947913 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:33.995450 9601 raft_consensus.cc:2468] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:33.935061 10459 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:33.969201 10561 raft_consensus.cc:493] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:33.988073 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 346) took 2955 ms. Trace:
I20260503 13:58:33.951750 10564 raft_consensus.cc:2749] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:34.001482 9642 consensus_queue.cc:260] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:33.973040 10543 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:33.996112 9643 ts_tablet_manager.cc:1403] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.797s user 0.005s sys 0.018s
I20260503 13:58:33.973868 10649 raft_consensus.cc:3060] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
W20260503 13:58:34.013187 10569 consensus_meta.cc:287] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.516s user 0.008s sys 0.000s
I20260503 13:58:34.030747 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:34.031378 9526 raft_consensus.cc:2468] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:33.993330 10647 raft_consensus.cc:2804] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:33.993592 10639 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:34.002135 10641 raft_consensus.cc:740] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:33.999750 9525 rpcz_store.cc:276] 0503 13:58:32.671984 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.666817 (+994833us) service_pool.cc:224] Handling call
0503 13:58:33.667161 (+ 344us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:33.796393 (+129232us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:33.796421 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:33.796932 (+ 511us) log.cc:844] Serialized 52 byte log entry
0503 13:58:33.848929 (+ 51997us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:33.848961 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.849019 (+ 58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:33.973586 (+124567us) raft_consensus.cc:1685] finished
0503 13:58:33.973605 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:33.973721 (+ 116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8568,"raft.queue_time_us":48637,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"thread_start_us":156549,"threads_started":2,"wal-append.queue_time_us":92151}
I20260503 13:58:34.037375 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:34.038421 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.015203 9644 ts_tablet_manager.cc:1434] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.101s user 0.005s sys 0.001s
I20260503 13:58:33.989748 10463 raft_consensus.cc:2749] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:34.009873 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:33.944880 10486 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.003585 10170 raft_consensus.cc:3060] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:33.951313 10666 consensus_queue.cc:1059] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20260503 13:58:34.005966 10475 raft_consensus.cc:2804] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:34.052213 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:33.934032 10634 consensus_queue.cc:237] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.030207 10243 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:34.058223 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 173999us
I20260503 13:58:34.059010 9416 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.040485 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.044531 10283 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:34.018517 10561 raft_consensus.cc:515] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:34.053648 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 130200us
W20260503 13:58:34.035167 9418 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:34.043448 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:34.040987 10580 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:33.993089 10560 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.033855 10569 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.072651 9416 leader_election.cc:400] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997) with higher term. Message: Invalid argument: T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4.
I20260503 13:58:34.073228 9416 leader_election.cc:403] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20260503 13:58:34.056592 10656 log.cc:927] Time spent T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.175s user 0.001s sys 0.000s
I20260503 13:58:34.069617 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.071527 9642 ts_tablet_manager.cc:1434] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.187s user 0.007s sys 0.000s
I20260503 13:58:34.052285 10475 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:34.069630 9525 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:34.074707 9600 raft_consensus.cc:2468] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:34.070011 9492 leader_election.cc:304] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.079512 10667 raft_consensus.cc:2804] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:34.079903 10667 raft_consensus.cc:493] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:34.080224 10667 raft_consensus.cc:3060] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:34.081756 9492 leader_election.cc:304] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.024703 9528 raft_consensus.cc:2468] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:34.084451 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.085011 9528 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:34.086614 10667 raft_consensus.cc:515] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.087579 10336 raft_consensus.cc:2749] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:34.088447 10667 leader_election.cc:290] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:34.089421 10506 consensus_meta.cc:287] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.840s user 0.006s sys 0.002s
I20260503 13:58:34.089758 9492 leader_election.cc:304] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.090487 10667 raft_consensus.cc:2804] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:34.090821 10667 raft_consensus.cc:493] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:34.091189 10667 raft_consensus.cc:3060] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:34.090821 9492 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.096176 9527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 406) took 1587 ms. Trace:
I20260503 13:58:34.077277 9416 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.096441 10336 raft_consensus.cc:2804] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:34.068590 10253 consensus_queue.cc:237] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.066677 10660 raft_consensus.cc:2749] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:34.071219 10641 consensus_queue.cc:260] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.017117 9594 rpcz_store.cc:276] 0503 13:58:31.027732 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.882167 (+2854435us) service_pool.cc:224] Handling call
0503 13:58:33.924779 (+ 42612us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.947352 (+ 22573us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.982583 (+ 35231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.107918 9416 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.106189 9492 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.109274 9528 raft_consensus.cc:2468] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:34.129172 9492 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.053426 10616 consensus_queue.cc:260] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.104082 9593 raft_consensus.cc:2368] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2.
I20260503 13:58:34.128832 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:34.136582 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 430) took 1955 ms. Trace:
I20260503 13:58:34.136029 9492 leader_election.cc:304] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.138406 9642 tablet_bootstrap.cc:492] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:34.138335 9445 rpcz_store.cc:276] 0503 13:58:32.180710 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.459651 (+1278941us) service_pool.cc:224] Handling call
0503 13:58:33.555824 (+ 96173us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.558381 (+ 2557us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.136366 (+577985us) spinlock_profiling.cc:243] Waited 95.9 ms on lock 0x7b540029f250. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:34.136501 (+ 135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":220514560}
I20260503 13:58:34.137575 10646 raft_consensus.cc:2804] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:34.139564 10646 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:34.139801 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.140069 9492 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.097141 10653 raft_consensus.cc:2749] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:34.128583 9416 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.137784 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.129256 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:34.141389 9416 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:34.097110 10372 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.138437 9528 raft_consensus.cc:2468] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:34.142616 9492 leader_election.cc:304] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.139920 10646 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
W20260503 13:58:34.139739 10385 consensus_meta.cc:287] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.585s user 0.007s sys 0.001s
I20260503 13:58:34.127863 10668 raft_consensus.cc:2749] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260503 13:58:34.075644 10445 raft_consensus.cc:2749] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997) with higher term. Message: Invalid argument: T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4.
W20260503 13:58:34.147089 10665 log.cc:927] Time spent T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
W20260503 13:58:34.147617 10626 log.cc:927] Time spent T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.570s user 0.000s sys 0.004s
I20260503 13:58:34.134873 10666 raft_consensus.cc:2749] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:34.096881 10336 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:34.101570 10561 leader_election.cc:290] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:34.101892 10544 consensus_meta.cc:287] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.863s user 0.008s sys 0.002s
I20260503 13:58:34.149097 10644 raft_consensus.cc:2749] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:34.096534 9527 rpcz_store.cc:276] 0503 13:58:32.509010 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.602585 (+1093575us) service_pool.cc:224] Handling call
0503 13:58:33.641459 (+ 38874us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:33.737175 (+ 95716us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:33.737202 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:33.775165 (+ 37963us) log.cc:844] Serialized 26 byte log entry
0503 13:58:33.983878 (+208713us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:33.983925 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:33.983991 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:34.095988 (+111997us) raft_consensus.cc:1685] finished
0503 13:58:34.096008 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:34.096124 (+ 116us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":72058,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":10657,"thread_start_us":281175,"threads_started":2,"wal-append.queue_time_us":238420}
I20260503 13:58:34.151954 10361 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.143981 10660 raft_consensus.cc:2749] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:34.148581 10633 raft_consensus.cc:2749] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:34.154778 9526 raft_consensus.cc:1240] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:34.156414 9492 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:34.104547 10475 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:34.174523 10544 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.159164 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:34.142696 10641 raft_consensus.cc:3060] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:34.140571 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:34.178371 9447 raft_consensus.cc:1240] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:34.178936 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 446) took 1834 ms. Trace:
I20260503 13:58:34.179234 9447 rpcz_store.cc:276] 0503 13:58:32.344917 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.177912 (+1832995us) service_pool.cc:224] Handling call
0503 13:58:34.178208 (+ 296us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.178728 (+ 520us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.178886 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.156186 10569 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.155813 10385 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.158098 10629 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.089792 10506 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.180914 10668 consensus_queue.cc:1059] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260503 13:58:34.181996 10668 consensus_queue.cc:1243] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer 4bea375a55034750a1f8804b0db97868 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260503 13:58:34.182274 10629 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.162245 10636 raft_consensus.cc:2804] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:34.091143 10632 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.182507 10634 raft_consensus.cc:2804] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:34.164733 9643 raft_consensus.cc:359] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.183298 9643 raft_consensus.cc:385] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:34.183573 9643 raft_consensus.cc:740] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:34.151167 10587 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:34.153167 10534 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:34.184158 10632 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.184739 10534 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:34.169241 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 473) took 1181 ms. Trace:
I20260503 13:58:34.184381 10636 raft_consensus.cc:493] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:34.169476 9492 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.185716 10636 raft_consensus.cc:3060] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:34.176331 10544 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.186544 9602 raft_consensus.cc:1240] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:34.186563 9492 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.186978 10506 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:34.187466 10629 raft_consensus.cc:2749] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:34.184381 10587 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:34.188134 10587 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:34.184194 9643 consensus_queue.cc:260] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.185204 9526 rpcz_store.cc:276] 0503 13:58:32.987710 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.142421 (+1154711us) service_pool.cc:224] Handling call
0503 13:58:34.154567 (+ 12146us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:34.169002 (+ 14435us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.169191 (+ 189us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.182441 10668 raft_consensus.cc:3055] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:34.157634 10616 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:34.162683 10667 raft_consensus.cc:515] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.198473 10544 consensus_queue.cc:1048] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:34.196204 10634 raft_consensus.cc:697] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:34.207877 9643 ts_tablet_manager.cc:1434] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.188s user 0.004s sys 0.002s
I20260503 13:58:34.201095 9644 tablet_bootstrap.cc:492] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:34.166662 10336 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:34.206588 10634 consensus_queue.cc:237] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.194100 10668 raft_consensus.cc:740] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:34.222692 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 385) took 1531 ms. Trace:
I20260503 13:58:34.203913 10649 raft_consensus.cc:515] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.223052 9598 rpcz_store.cc:276] 0503 13:58:32.691509 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:32.925937 (+234428us) service_pool.cc:224] Handling call
0503 13:58:34.222494 (+1296557us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b5400058554. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:34.222636 (+ 142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2061402496}
W20260503 13:58:34.229447 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:34.229673 9643 tablet_bootstrap.cc:492] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:34.225298 9418 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:34.164904 10653 consensus_queue.cc:1059] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.045s
I20260503 13:58:34.181233 10625 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.233651 9567 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
W20260503 13:58:34.216842 10638 consensus_meta.cc:287] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.513s user 0.006s sys 0.002s
I20260503 13:58:34.230538 9418 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.219998 10486 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:34.232982 10544 raft_consensus.cc:2804] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:34.238780 10544 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:34.239147 10544 raft_consensus.cc:3060] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:34.244407 9418 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:34.245242 9418 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:34.246168 9445 raft_consensus.cc:2393] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 6: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:34.246376 9418 leader_election.cc:304] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters: 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.245808 10625 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.248489 10660 raft_consensus.cc:2804] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:34.248796 10445 raft_consensus.cc:2804] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:34.253167 10445 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:34.253499 10445 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:34.237564 10560 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.237056 10638 raft_consensus.cc:2749] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:34.251943 10660 raft_consensus.cc:697] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:34.249545 10506 consensus_queue.cc:1059] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260503 13:58:34.246860 10283 raft_consensus.cc:2804] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:34.250433 10542 raft_consensus.cc:2804] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260503 13:58:34.259544 10671 log.cc:927] Time spent T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20260503 13:58:34.265445 10653 raft_consensus.cc:2749] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:34.262405 10667 leader_election.cc:290] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.278761 9519 raft_consensus.cc:1240] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:34.289371 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 454) took 1632 ms. Trace:
I20260503 13:58:34.289811 9447 rpcz_store.cc:276] 0503 13:58:32.656756 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.179920 (+1523164us) service_pool.cc:224] Handling call
0503 13:58:34.180180 (+ 260us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.180366 (+ 186us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:34.204013 (+ 23647us) log.cc:844] Serialized 14 byte log entry
0503 13:58:34.284023 (+ 80010us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:34.289094 (+ 5071us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:34.289139 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.289204 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:34.289316 (+ 112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3302,"prepare.queue_time_us":156,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":7963,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":2994,"thread_start_us":63682,"threads_started":1,"wal-append.queue_time_us":71341}
I20260503 13:58:34.291687 9522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 470) took 1341 ms. Trace:
I20260503 13:58:34.274070 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 437) took 2065 ms. Trace:
I20260503 13:58:34.252667 10475 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.279204 10542 raft_consensus.cc:493] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:34.301438 10542 raft_consensus.cc:3060] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:34.303598 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 347) took 3210 ms. Trace:
I20260503 13:58:34.309966 9598 rpcz_store.cc:276] 0503 13:58:31.093456 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.229507 (+3136051us) service_pool.cc:224] Handling call
0503 13:58:34.249953 (+ 20446us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:34.255461 (+ 5508us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:34.255576 (+ 115us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:34.255996 (+ 420us) log.cc:844] Serialized 26 byte log entry
0503 13:58:34.301636 (+ 45640us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:34.301679 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.301747 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:34.303418 (+ 1671us) raft_consensus.cc:1685] finished
0503 13:58:34.303436 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:34.303549 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8663,"raft.queue_time_us":9838,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":10648,"thread_start_us":18740,"threads_started":1,"wal-append.queue_time_us":28879}
I20260503 13:58:34.298888 10283 raft_consensus.cc:697] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:34.292672 10629 raft_consensus.cc:2749] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:34.315553 9566 leader_election.cc:400] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599) with higher term. Message: Invalid argument: T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2.
I20260503 13:58:34.323669 9566 leader_election.cc:403] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Cancelling election due to peer responding with higher term
I20260503 13:58:34.324723 9566 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:34.317678 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.326433 9598 raft_consensus.cc:1240] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260503 13:58:34.318922 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:34.327033 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 348) took 3004 ms. Trace:
I20260503 13:58:34.326455 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.327740 9445 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:34.327554 9598 rpcz_store.cc:276] 0503 13:58:31.322957 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.311858 (+2988901us) service_pool.cc:224] Handling call
0503 13:58:34.326235 (+ 14377us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:34.326816 (+ 581us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.326988 (+ 172us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.331722 10646 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.252655 10668 consensus_queue.cc:260] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.333906 9642 tablet_bootstrap.cc:654] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:34.365551 9445 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:34.317062 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.333120 10580 raft_consensus.cc:3060] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
@ 0x7f9f41f35980 (unknown) at ??:0
I20260503 13:58:34.373277 9602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 360) took 3606 ms. Trace:
I20260503 13:58:34.373411 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.373837 9602 rpcz_store.cc:276] 0503 13:58:30.766375 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.705290 (+2938915us) service_pool.cc:224] Handling call
0503 13:58:33.725031 (+ 19741us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.186930 (+461899us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.373072 (+186142us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b54002585d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:34.373226 (+ 154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1060986112}
I20260503 13:58:34.374549 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.375268 9445 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:34.377508 10646 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.378312 10385 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.379091 9602 raft_consensus.cc:1240] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:34.380266 9602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 351) took 2868 ms. Trace:
I20260503 13:58:34.380612 9602 rpcz_store.cc:276] 0503 13:58:31.512132 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.374718 (+2862586us) service_pool.cc:224] Handling call
0503 13:58:34.378940 (+ 4222us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.380048 (+ 1108us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.380212 (+ 164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.341194 10561 raft_consensus.cc:2749] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:34.379619 9566 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:34.378557 9594 raft_consensus.cc:2468] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:34.317356 10636 raft_consensus.cc:515] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.312966 10620 raft_consensus.cc:515] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.365936 10649 leader_election.cc:290] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.363422 10179 consensus_queue.cc:1059] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.023s
W20260503 13:58:34.388031 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:34.296470 9522 rpcz_store.cc:276] 0503 13:58:32.950767 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.982321 (+1031554us) service_pool.cc:224] Handling call
0503 13:58:34.014621 (+ 32300us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.034461 (+ 19840us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:34.058205 (+ 23744us) log.cc:844] Serialized 14 byte log entry
0503 13:58:34.283005 (+224800us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:34.285636 (+ 2631us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:34.291435 (+ 5799us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.291514 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:34.291635 (+ 121us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":5114,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":19085,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":51,"thread_start_us":199194,"threads_started":2,"wal-append.queue_time_us":128447}
I20260503 13:58:34.395905 9445 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 6.
I20260503 13:58:34.408622 9566 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:34.379727 10295 consensus_queue.cc:1059] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
I20260503 13:58:34.405536 9594 raft_consensus.cc:1275] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260503 13:58:34.409821 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 354) took 1938 ms. Trace:
I20260503 13:58:34.410163 9594 rpcz_store.cc:276] 0503 13:58:32.471745 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.404910 (+1933165us) service_pool.cc:224] Handling call
0503 13:58:34.405220 (+ 310us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.409561 (+ 4341us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.409763 (+ 202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.410640 10295 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election won for term 6
I20260503 13:58:34.411089 9643 tablet_bootstrap.cc:654] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:34.411243 10561 raft_consensus.cc:3055] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:34.411351 9594 raft_consensus.cc:1240] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:34.366149 10537 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.408406 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.365995 10459 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.412904 9445 raft_consensus.cc:3060] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:34.414008 9526 raft_consensus.cc:1240] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:34.390241 10506 raft_consensus.cc:2749] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260503 13:58:34.415081 9602 raft_consensus.cc:1240] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:34.299149 9446 rpcz_store.cc:276] 0503 13:58:32.209029 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.484996 (+1275967us) service_pool.cc:224] Handling call
0503 13:58:33.513385 (+ 28389us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.554257 (+ 40872us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:33.585127 (+ 30870us) log.cc:844] Serialized 14 byte log entry
0503 13:58:34.206723 (+621596us) spinlock_profiling.cc:243] Waited 8.06 ms on lock 0x7b18001824a0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea70587 00007f9f47869bc9 00007f9f3ea8ccdb 00007f9f3ea8af7f 00007f9f3ea8d242 00007f9f416c45ab 00007f9f416ca0c3 00007f9f416ca5da 00007f9f41b35e0a
0503 13:58:34.263730 (+ 57007us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:34.263798 (+ 68us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:34.263843 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.263915 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:34.265502 (+ 1587us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3867,"prepare.queue_time_us":13408,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":2724,"raft.queue_time_us":19193,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":3935,"spinlock_wait_cycles":18549120,"thread_start_us":44512,"threads_started":1,"wal-append.queue_time_us":728221}
I20260503 13:58:34.411657 10561 raft_consensus.cc:740] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:34.416991 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 366) took 1942 ms. Trace:
I20260503 13:58:34.417443 10561 consensus_queue.cc:260] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.418613 10561 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:34.419214 10620 leader_election.cc:290] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.419912 9644 tablet_bootstrap.cc:654] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:34.411121 10295 raft_consensus.cc:697] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 6 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:34.421070 10358 raft_consensus.cc:3055] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:34.362766 10660 consensus_queue.cc:237] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.421659 10636 leader_election.cc:290] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.421404 10295 consensus_queue.cc:237] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.422997 9523 raft_consensus.cc:1240] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:34.423621 9523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 468) took 1522 ms. Trace:
I20260503 13:58:34.423914 9523 rpcz_store.cc:276] 0503 13:58:32.901438 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.962746 (+1061308us) service_pool.cc:224] Handling call
0503 13:58:34.422647 (+459901us) spinlock_profiling.cc:243] Waited 3.18 ms on lock 0x7b5800004598. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:34.422831 (+ 184us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.423356 (+ 525us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.423559 (+ 203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7315328}
I20260503 13:58:34.400981 10283 consensus_queue.cc:237] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.405524 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.406050 10655 raft_consensus.cc:2804] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260503 13:58:34.427911 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 64588.5us
W20260503 13:58:34.426949 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 63209us
I20260503 13:58:34.406433 10475 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:34.429230 9492 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:34.421425 10358 raft_consensus.cc:740] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:34.434075 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.434964 10358 consensus_queue.cc:260] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.435540 10639 raft_consensus.cc:2749] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:34.439872 10544 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.431376 9522 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:34.437029 10537 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.430460 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.441715 10544 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.441843 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.442366 9447 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:34.437443 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:34.425568 10543 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.430608 10506 raft_consensus.cc:493] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:34.432541 10636 raft_consensus.cc:2804] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:34.417394 9594 rpcz_store.cc:276] 0503 13:58:32.474908 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.410899 (+1935991us) service_pool.cc:224] Handling call
0503 13:58:34.411186 (+ 287us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.411734 (+ 548us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.415896 (+ 4162us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.467451 10506 raft_consensus.cc:515] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.470160 10534 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.471410 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.472077 9523 raft_consensus.cc:2393] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 6: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:34.468739 9528 raft_consensus.cc:1240] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260503 13:58:34.472869 10534 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.473354 10636 raft_consensus.cc:697] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:34.474291 10636 consensus_queue.cc:237] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.473307 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.476442 10544 raft_consensus.cc:493] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:34.477456 10544 raft_consensus.cc:515] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.485947 9447 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:34.475773 10358 raft_consensus.cc:3060] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:34.486806 9492 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:34.493660 10639 raft_consensus.cc:2804] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election won for term 6
I20260503 13:58:34.494086 10639 raft_consensus.cc:697] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 6 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:34.494810 10639 consensus_queue.cc:237] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.488994 9445 raft_consensus.cc:2468] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:34.498798 9492 leader_election.cc:304] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:34.499622 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:34.491029 9418 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:34.478113 10542 raft_consensus.cc:515] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.489471 9522 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 6.
I20260503 13:58:34.500192 10445 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.503638 10649 raft_consensus.cc:2804] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:34.503700 9567 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:34.504199 10649 raft_consensus.cc:697] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:34.504284 10445 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:34.506150 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:34.500238 9445 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:34.489024 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.492973 9523 raft_consensus.cc:2468] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:34.497601 10580 raft_consensus.cc:2749] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: Vote denied by peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599) with higher term. Message: Invalid argument: T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2.
I20260503 13:58:34.507133 9447 raft_consensus.cc:2393] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 6: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:34.507418 9418 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:34.508857 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.505084 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.511569 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.509974 9523 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:34.512149 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:34.515363 9444 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:34.515636 10459 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:34.505019 10649 consensus_queue.cc:237] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.516232 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:34.516712 9594 raft_consensus.cc:3060] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:34.517179 10243 raft_consensus.cc:3055] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:34.517513 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:34.518280 9566 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:34.512960 10655 raft_consensus.cc:493] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:34.513233 10544 leader_election.cc:290] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.514792 10543 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.489892 10635 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.518522 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.524396 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030fcc1b19c3489ea0ca320576ed54ee" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:34.525141 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 376) took 2274 ms. Trace:
I20260503 13:58:34.545568 9453 rpcz_store.cc:276] 0503 13:58:32.250517 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.638652 (+1388135us) service_pool.cc:224] Handling call
0503 13:58:33.796349 (+157697us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.808921 (+ 12572us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.524977 (+716056us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b5400294ad0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:34.525092 (+ 115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":361936256}
I20260503 13:58:34.546720 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:34.547652 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.548513 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:34.526393 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:34.527681 9451 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 435) took 2320 ms (client timeout 3037 ms). Trace:
W20260503 13:58:34.549563 9451 rpcz_store.cc:269] 0503 13:58:32.207537 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.259888 (+1052351us) service_pool.cc:224] Handling call
0503 13:58:34.527494 (+1267606us) spinlock_profiling.cc:243] Waited 479 ms on lock 0x7b540007fdd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:34.527623 (+ 129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1102625536}
I20260503 13:58:34.519114 10347 raft_consensus.cc:2804] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:34.525379 10445 raft_consensus.cc:2804] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:34.517537 10243 raft_consensus.cc:740] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:34.550769 9566 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:34.538805 9491 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.541749 9523 raft_consensus.cc:2468] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:34.509305 10506 leader_election.cc:290] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:34.520180 10616 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election won for term 6
I20260503 13:58:34.502307 10336 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.553689 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:34.579643 9599 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 388) took 1887 ms (client timeout 2165 ms). Trace:
I20260503 13:58:34.580441 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.580775 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.539953 10655 raft_consensus.cc:3060] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:34.553917 9492 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:34.565304 9444 raft_consensus.cc:2393] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:34.565567 9452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 434) took 2367 ms. Trace:
I20260503 13:58:34.579152 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.551815 9593 raft_consensus.cc:1240] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:34.566519 10251 raft_consensus.cc:2749] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:34.545118 10537 consensus_queue.cc:1059] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260503 13:58:34.567287 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:34.587565 9451 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:34.587975 9597 raft_consensus.cc:2468] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:34.589040 9597 raft_consensus.cc:1240] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260503 13:58:34.589423 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 374) took 1731 ms. Trace:
I20260503 13:58:34.589602 9597 rpcz_store.cc:276] 0503 13:58:32.858281 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.588523 (+1730242us) service_pool.cc:224] Handling call
0503 13:58:34.588838 (+ 315us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:34.589272 (+ 434us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.589391 (+ 119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.590297 9597 raft_consensus.cc:1240] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:34.590662 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 387) took 1480 ms. Trace:
I20260503 13:58:34.590845 9597 rpcz_store.cc:276] 0503 13:58:33.110373 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.590019 (+1479646us) service_pool.cc:224] Handling call
0503 13:58:34.590200 (+ 181us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.590487 (+ 287us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.590627 (+ 140us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.551903 10347 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:34.568832 9519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 414) took 1688 ms. Trace:
I20260503 13:58:34.554649 9567 leader_election.cc:304] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:34.563434 10580 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:34.574432 9524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 390) took 2162 ms. Trace:
I20260503 13:58:34.606208 9524 rpcz_store.cc:276] 0503 13:58:32.411714 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.098569 (+686855us) service_pool.cc:224] Handling call
0503 13:58:33.098785 (+ 216us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.964964 (+866179us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.574222 (+609258us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b54003d4ad4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:34.574375 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1835934080}
I20260503 13:58:34.581146 9453 raft_consensus.cc:3060] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:34.554369 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:34.580018 9599 rpcz_store.cc:269] 0503 13:58:32.692220 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.030279 (+338059us) service_pool.cc:224] Handling call
0503 13:58:34.579465 (+1549186us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b5400260cd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:34.579591 (+ 126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1705468672}
I20260503 13:58:34.528708 10639 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
W20260503 13:58:34.586393 9418 leader_election.cc:343] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:34.573714 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.586921 10616 raft_consensus.cc:697] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 6 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:34.592247 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:34.592753 10336 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.571401 9491 leader_election.cc:400] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Vote denied by peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997) with higher term. Message: Invalid argument: T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 5. Current term is 6.
I20260503 13:58:34.618057 9491 leader_election.cc:403] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260503 13:58:34.593345 9519 rpcz_store.cc:276] 0503 13:58:32.880061 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.647797 (+767736us) service_pool.cc:224] Handling call
0503 13:58:33.676351 (+ 28554us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.279180 (+602829us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.568601 (+289421us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x7b54003d4fd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:34.568763 (+ 162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1376760576}
I20260503 13:58:34.617501 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:34.592834 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:34.597226 9445 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:34.573312 10243 consensus_queue.cc:260] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.573534 10445 raft_consensus.cc:697] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:34.598505 10347 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:34.598999 10537 raft_consensus.cc:3055] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:34.599879 9593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 373) took 1746 ms. Trace:
I20260503 13:58:34.600308 9595 raft_consensus.cc:1240] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
W20260503 13:58:34.600538 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 45974us
I20260503 13:58:34.587406 9452 rpcz_store.cc:276] 0503 13:58:32.197831 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.473932 (+1276101us) service_pool.cc:224] Handling call
0503 13:58:33.502951 (+ 29019us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.973955 (+471004us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.565343 (+591388us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b54001d33d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:34.565501 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":898799488}
I20260503 13:58:34.601583 10506 raft_consensus.cc:2804] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:34.636135 9451 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:34.636391 10506 raft_consensus.cc:697] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:34.637151 9492 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:34.637292 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:34.602604 10693 log.cc:927] Time spent T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260503 13:58:34.637830 10676 raft_consensus.cc:493] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:34.638012 10646 raft_consensus.cc:2804] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election won for term 6
I20260503 13:58:34.638458 10646 raft_consensus.cc:697] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 6 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:34.638247 10676 raft_consensus.cc:515] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.608661 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:34.585693 10688 log.cc:927] Time spent T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20260503 13:58:34.616185 9453 raft_consensus.cc:2468] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:34.642042 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:34.545190 10640 consensus_queue.cc:1048] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20260503 13:58:34.573328 10647 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:34.619838 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.620048 9447 raft_consensus.cc:2410] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 4 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260503 13:58:34.594970 10543 raft_consensus.cc:2749] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:34.651261 9644 tablet_bootstrap.cc:492] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:34.634950 10336 raft_consensus.cc:2749] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997) with higher term. Message: Invalid argument: T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 5. Current term is 6.
I20260503 13:58:34.623490 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.623864 9643 tablet_bootstrap.cc:492] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
W20260503 13:58:34.630461 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 66273.8us
I20260503 13:58:34.564306 10650 consensus_queue.cc:1059] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.045s
W20260503 13:58:34.619117 10547 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.020s sys 0.029s
I20260503 13:58:34.612432 10653 consensus_queue.cc:1048] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260503 13:58:34.612571 10635 consensus_queue.cc:1059] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 0, Time since last communication: 0.022s
W20260503 13:58:34.659519 10678 log.cc:927] Time spent T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.247s user 0.003s sys 0.000s
I20260503 13:58:34.649096 10647 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:34.640097 9524 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:34.660801 9527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 477) took 1395 ms. Trace:
I20260503 13:58:34.643283 9492 leader_election.cc:304] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:34.644304 9523 raft_consensus.cc:2393] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:34.636332 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:34.629990 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.631719 10537 raft_consensus.cc:740] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:34.611807 10639 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.674245 10639 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.674829 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 479) took 1273 ms. Trace:
I20260503 13:58:34.675186 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 476) took 1454 ms. Trace:
I20260503 13:58:34.659464 10635 raft_consensus.cc:3055] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:34.675525 9596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 380) took 1772 ms. Trace:
I20260503 13:58:34.675784 10635 raft_consensus.cc:740] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:34.675928 9596 rpcz_store.cc:276] 0503 13:58:32.902905 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.497451 (+594546us) service_pool.cc:224] Handling call
0503 13:58:34.675361 (+1177910us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x7b540031b5d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:34.675467 (+ 106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":908623232}
I20260503 13:58:34.677069 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:34.677279 9567 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:34.677698 9522 raft_consensus.cc:2468] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
W20260503 13:58:34.677819 9600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 402) took 1367 ms (client timeout 1643 ms). Trace:
I20260503 13:58:34.649755 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:34.678248 9600 rpcz_store.cc:269] 0503 13:58:33.310175 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.329047 (+ 18872us) service_pool.cc:224] Handling call
0503 13:58:34.677654 (+1348607us) spinlock_profiling.cc:243] Waited 705 ms on lock 0x7b54002bacd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:34.677762 (+ 108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1621184896}
I20260503 13:58:34.679360 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:34.680145 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:34.680465 9600 raft_consensus.cc:3060] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:34.680619 9598 raft_consensus.cc:1240] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:34.680927 9567 leader_election.cc:304] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:34.675194 9526 rpcz_store.cc:276] 0503 13:58:33.401697 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.190778 (+789081us) service_pool.cc:224] Handling call
0503 13:58:34.191025 (+ 247us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.416261 (+225236us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.674587 (+258326us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b54000c67d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:34.674769 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":512414592}
I20260503 13:58:34.682718 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.675529 9528 rpcz_store.cc:276] 0503 13:58:33.222350 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.145792 (+923442us) service_pool.cc:224] Handling call
0503 13:58:34.149066 (+ 3274us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:34.472663 (+323597us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.674977 (+202314us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b54003d45d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:34.675134 (+ 157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":734673152}
I20260503 13:58:34.684226 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.651743 9644 ts_tablet_manager.cc:1403] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.451s user 0.028s sys 0.019s
I20260503 13:58:34.622018 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.621268 9566 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:34.652850 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 459) took 1732 ms. Trace:
I20260503 13:58:34.653386 9643 ts_tablet_manager.cc:1403] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.429s user 0.006s sys 0.011s
I20260503 13:58:34.653136 9445 raft_consensus.cc:2393] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 6: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:34.653961 9451 raft_consensus.cc:1240] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:34.654296 10676 leader_election.cc:290] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:34.648751 10687 log.cc:927] Time spent T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.183s user 0.001s sys 0.000s
I20260503 13:58:34.698247 9449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 427) took 2518 ms. Trace:
I20260503 13:58:34.698024 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 455) took 1841 ms. Trace:
I20260503 13:58:34.658391 9447 raft_consensus.cc:1240] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260503 13:58:34.662925 10547 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s user 0.020s sys 0.030s
I20260503 13:58:34.651631 10616 consensus_queue.cc:237] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.660699 10634 consensus_queue.cc:1059] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260503 13:58:34.702720 10634 raft_consensus.cc:3055] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:34.703042 10634 raft_consensus.cc:740] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:34.663537 9444 raft_consensus.cc:1240] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260503 13:58:34.703811 10634 consensus_queue.cc:260] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 1, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.704824 10634 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:34.705264 9444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 405) took 1759 ms. Trace:
I20260503 13:58:34.663681 10558 raft_consensus.cc:2804] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:34.632251 10587 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.639195 10646 consensus_queue.cc:237] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.666250 10542 leader_election.cc:290] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:34.668051 9642 tablet_bootstrap.cc:492] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:34.711053 9642 ts_tablet_manager.cc:1403] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.597s user 0.013s sys 0.005s
I20260503 13:58:34.661162 9527 rpcz_store.cc:276] 0503 13:58:33.265802 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.181346 (+915544us) service_pool.cc:224] Handling call
0503 13:58:34.181661 (+ 315us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:34.183429 (+ 1768us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:34.183457 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:34.183918 (+ 461us) log.cc:844] Serialized 26 byte log entry
0503 13:58:34.310793 (+126875us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:34.324390 (+ 13597us) log.cc:844] Serialized 14 byte log entry
0503 13:58:34.324598 (+ 208us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.324663 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:34.503499 (+178836us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b18003047e0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea8db62 00007f9f3ea8fe01 00007f9f3ea8fdb9 00007f9f3ea8fd49 00007f9f3ea8fd11 00007f9f3ea8f00d 0000000000379254 0000000000379099 00007f9f3ea70fba
0503 13:58:34.659060 (+155561us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:34.660606 (+ 1546us) raft_consensus.cc:1685] finished
0503 13:58:34.660630 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:34.660744 (+ 114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2669,"prepare.queue_time_us":15075,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":1381,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":4334,"spinlock_wait_cycles":45688832,"thread_start_us":126549,"threads_started":1,"wal-append.queue_time_us":179859}
W20260503 13:58:34.670147 9601 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 385) took 1628 ms (client timeout 1525 ms). Trace:
W20260503 13:58:34.711848 9601 rpcz_store.cc:269] 0503 13:58:33.041595 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.091186 (+ 49591us) service_pool.cc:224] Handling call
0503 13:58:34.669999 (+1578813us) spinlock_profiling.cc:243] Waited 816 ms on lock 0x7b5400058f54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:34.670087 (+ 88us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1877094144}
I20260503 13:58:34.713047 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:34.713755 9601 raft_consensus.cc:2393] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:34.671567 9595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 338) took 3904 ms. Trace:
I20260503 13:58:34.632172 9593 rpcz_store.cc:276] 0503 13:58:32.853935 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.524211 (+1670276us) service_pool.cc:224] Handling call
0503 13:58:34.524494 (+ 283us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.599610 (+ 75116us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.599820 (+ 210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.684724 9528 raft_consensus.cc:2393] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
W20260503 13:58:34.716341 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:34.697809 9519 raft_consensus.cc:2393] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 6: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:34.654407 9491 leader_election.cc:400] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Vote denied by peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997) with higher term. Message: Invalid argument: T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2.
I20260503 13:58:34.698590 9449 rpcz_store.cc:276] 0503 13:58:32.179985 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.455060 (+1275075us) service_pool.cc:224] Handling call
0503 13:58:33.455322 (+ 262us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:33.891754 (+436432us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.696550 (+804796us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b54001a1154. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:34.698196 (+ 1646us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":964813056}
I20260503 13:58:34.716804 9491 leader_election.cc:403] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Cancelling election due to peer responding with higher term
W20260503 13:58:34.717422 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 63129.7us
W20260503 13:58:34.662462 9492 leader_election.cc:343] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:34.717976 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 55550.5us
I20260503 13:58:34.718667 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:34.719667 10676 raft_consensus.cc:3060] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 2
I20260503 13:58:34.676990 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:34.699412 9451 rpcz_store.cc:276] 0503 13:58:32.857021 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.640280 (+1783259us) service_pool.cc:224] Handling call
0503 13:58:34.652955 (+ 12675us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.697809 (+ 44854us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.697975 (+ 166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.678224 10193 raft_consensus.cc:2749] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260503 13:58:34.724231 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 480) took 1221 ms. Trace:
I20260503 13:58:34.674853 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.701188 9445 raft_consensus.cc:1240] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:34.690572 9524 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:34.697400 9453 rpcz_store.cc:276] 0503 13:58:32.917896 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.649594 (+1731698us) service_pool.cc:224] Handling call
0503 13:58:34.649867 (+ 273us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.650020 (+ 153us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:34.650044 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:34.650086 (+ 42us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:34.650120 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.650171 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:34.650259 (+ 88us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.724629 9528 rpcz_store.cc:276] 0503 13:58:33.503034 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.723535 (+1220501us) service_pool.cc:224] Handling call
0503 13:58:34.723794 (+ 259us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.723954 (+ 160us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:34.723977 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:34.724016 (+ 39us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:34.724045 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.724094 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:34.724183 (+ 89us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.725551 9453 raft_consensus.cc:1240] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
W20260503 13:58:34.707052 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 85871.6us
I20260503 13:58:34.700042 10564 consensus_queue.cc:1059] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260503 13:58:34.709687 10283 raft_consensus.cc:2749] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:34.708129 10475 consensus_queue.cc:1059] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260503 13:58:34.698179 10295 raft_consensus.cc:2804] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:34.717485 9449 raft_consensus.cc:1240] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260503 13:58:34.768487 9449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 460) took 1811 ms. Trace:
I20260503 13:58:34.699069 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 456) took 1839 ms. Trace:
I20260503 13:58:34.768891 9418 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:34.769068 9447 rpcz_store.cc:276] 0503 13:58:32.859979 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.649127 (+1789148us) service_pool.cc:224] Handling call
0503 13:58:34.658252 (+ 9125us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:34.698845 (+ 40593us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.699024 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.768822 9449 rpcz_store.cc:276] 0503 13:58:32.957448 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.717003 (+1759555us) service_pool.cc:224] Handling call
0503 13:58:34.717296 (+ 293us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:34.768266 (+ 50970us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.768443 (+ 177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.770880 9449 raft_consensus.cc:1240] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:34.722517 9596 raft_consensus.cc:2376] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4. Re-sending same reply.
I20260503 13:58:34.723354 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:34.664812 10170 consensus_queue.cc:1059] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260503 13:58:34.773589 9449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 464) took 1290 ms. Trace:
I20260503 13:58:34.773929 9449 rpcz_store.cc:276] 0503 13:58:33.483290 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.770590 (+1287300us) service_pool.cc:224] Handling call
0503 13:58:34.770778 (+ 188us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.773402 (+ 2624us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.773544 (+ 142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.637326 10506 consensus_queue.cc:237] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.706432 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:34.762950 9520 raft_consensus.cc:1240] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:34.762962 9528 raft_consensus.cc:1240] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:34.763583 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 463) took 1803 ms. Trace:
I20260503 13:58:34.775920 9453 rpcz_store.cc:276] 0503 13:58:32.960293 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.725133 (+1764840us) service_pool.cc:224] Handling call
0503 13:58:34.725387 (+ 254us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.725906 (+ 519us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.763525 (+ 37619us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:34.741319 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:34.764166 9519 raft_consensus.cc:1240] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260503 13:58:34.764348 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:34.778837 9601 raft_consensus.cc:2393] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:34.721033 9452 raft_consensus.cc:2393] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:34.765460 9566 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:34.764597 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 410) took 1817 ms. Trace:
I20260503 13:58:34.764019 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:34.783322 9527 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:34.719694 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:34.771627 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:34.762328 9448 raft_consensus.cc:2393] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:34.697182 10445 consensus_queue.cc:237] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.715758 10558 raft_consensus.cc:697] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:34.775506 9449 raft_consensus.cc:1240] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:34.775070 10283 raft_consensus.cc:3055] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:34.788832 9451 raft_consensus.cc:1240] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:34.789351 9449 raft_consensus.cc:1240] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260503 13:58:34.790028 9519 raft_consensus.cc:1240] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:34.789388 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 467) took 1303 ms. Trace:
I20260503 13:58:34.785013 9523 raft_consensus.cc:1240] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:34.779808 10475 raft_consensus.cc:3055] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:34.783957 9600 raft_consensus.cc:2468] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:34.705610 9444 rpcz_store.cc:276] 0503 13:58:32.945398 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.651668 (+1706270us) service_pool.cc:224] Handling call
0503 13:58:34.663300 (+ 11632us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:34.704387 (+ 41087us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.705209 (+ 822us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.791078 9447 raft_consensus.cc:1240] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260503 13:58:34.792944 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:34.793569 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 412) took 1800 ms. Trace:
I20260503 13:58:34.791347 9451 rpcz_store.cc:276] 0503 13:58:33.486190 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.771664 (+1285474us) service_pool.cc:224] Handling call
0503 13:58:34.788663 (+ 16999us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.789191 (+ 528us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.789341 (+ 150us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.772729 10295 raft_consensus.cc:493] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:34.793984 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:34.795907 9600 raft_consensus.cc:2393] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 5: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:34.790728 9449 raft_consensus.cc:1275] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:34.793881 9447 rpcz_store.cc:276] 0503 13:58:32.993447 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.769752 (+1776305us) service_pool.cc:224] Handling call
0503 13:58:34.790921 (+ 21169us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:34.793323 (+ 2402us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.793508 (+ 185us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.788501 9446 raft_consensus.cc:2393] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 6: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:34.780123 10542 raft_consensus.cc:2804] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:34.804010 9596 raft_consensus.cc:2468] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:34.796955 9445 rpcz_store.cc:276] 0503 13:58:32.946758 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.699058 (+1752300us) service_pool.cc:224] Handling call
0503 13:58:34.701051 (+ 1993us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:34.726690 (+ 25639us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.764527 (+ 37837us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.810621 9519 raft_consensus.cc:1240] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 6 LEADER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260503 13:58:34.808245 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030fcc1b19c3489ea0ca320576ed54ee" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:34.788527 10283 raft_consensus.cc:740] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:34.786921 10486 raft_consensus.cc:2804] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:34.725003 10537 consensus_queue.cc:260] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 1, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.787595 10193 consensus_queue.cc:1059] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 1, Time since last communication: 0.038s
W20260503 13:58:34.826112 9450 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 404) took 1897 ms (client timeout 1745 ms). Trace:
I20260503 13:58:34.826303 9447 raft_consensus.cc:1275] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260503 13:58:34.842913 9450 rpcz_store.cc:269] 0503 13:58:32.939072 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.091800 (+152728us) service_pool.cc:224] Handling call
0503 13:58:34.815318 (+1723518us) spinlock_profiling.cc:243] Waited 909 ms on lock 0x7b54001d2754. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:34.826056 (+ 10738us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2090067200}
I20260503 13:58:34.791962 10475 raft_consensus.cc:740] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:34.715583 9595 rpcz_store.cc:276] 0503 13:58:30.766672 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.739574 (+2972902us) service_pool.cc:224] Handling call
0503 13:58:34.585370 (+845796us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.634861 (+ 49491us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.671391 (+ 36530us) spinlock_profiling.cc:243] Waited 784 ms on lock 0x7b5400260cd0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:34.671511 (+ 120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1803752704}
@ 0x7f9f3b33be87 gsignal at ??:0
I20260503 13:58:34.839762 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 350) took 3374 ms. Trace:
I20260503 13:58:34.815810 9602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 365) took 2844 ms. Trace:
I20260503 13:58:35.032939 9643 raft_consensus.cc:359] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.033083 9602 rpcz_store.cc:276] 0503 13:58:31.971670 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.386853 (+2415183us) service_pool.cc:224] Handling call
0503 13:58:34.405914 (+ 19061us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.415408 (+ 9494us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.815604 (+400196us) spinlock_profiling.cc:243] Waited 9 ms on lock 0x7b54000abc54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:34.815745 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20705536}
I20260503 13:58:35.033607 9643 raft_consensus.cc:385] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:35.033874 9643 raft_consensus.cc:740] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:35.034272 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:35.034442 9643 consensus_queue.cc:260] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.034965 9602 raft_consensus.cc:2393] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 6: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:35.035645 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.036154 9643 ts_tablet_manager.cc:1434] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.339s user 0.000s sys 0.030s
W20260503 13:58:34.840713 9521 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37220 (request call id 481) took 1332 ms (client timeout 1625 ms). Trace:
I20260503 13:58:34.842190 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:34.799152 10295 raft_consensus.cc:3060] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:34.843065 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 474) took 1772 ms. Trace:
I20260503 13:58:35.061174 9525 rpcz_store.cc:276] 0503 13:58:33.070323 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.143614 (+1073291us) service_pool.cc:224] Handling call
0503 13:58:34.145746 (+ 2132us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:34.228050 (+ 82304us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:34.228081 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:34.228477 (+ 396us) log.cc:844] Serialized 26 byte log entry
0503 13:58:34.402592 (+174115us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:34.402633 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.402722 (+ 89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:34.655516 (+252794us) raft_consensus.cc:1685] finished
0503 13:58:34.655535 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:34.837131 (+181596us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b540027c9d0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:34.843011 (+ 5880us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":21658,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"spinlock_wait_cycles":2771712,"thread_start_us":253317,"threads_started":2,"wal-append.queue_time_us":203471}
I20260503 13:58:35.063045 9525 raft_consensus.cc:1240] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:35.063550 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 425) took 1076 ms. Trace:
I20260503 13:58:35.063830 9525 rpcz_store.cc:276] 0503 13:58:33.986743 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.062621 (+1075878us) service_pool.cc:224] Handling call
0503 13:58:35.062893 (+ 272us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:35.063349 (+ 456us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:35.063502 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:34.825517 9451 raft_consensus.cc:3055] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:35.064680 9451 raft_consensus.cc:740] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
W20260503 13:58:34.825327 10701 log.cc:927] Time spent T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
I20260503 13:58:34.864985 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:35.066650 9519 raft_consensus.cc:1240] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:35.067102 10170 consensus_queue.cc:1059] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:58:35.067652 9519 raft_consensus.cc:1240] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:35.067850 10170 raft_consensus.cc:3055] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:35.068162 10170 raft_consensus.cc:740] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:35.025624 10486 raft_consensus.cc:697] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:35.031800 10193 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:34.829613 10542 raft_consensus.cc:697] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:34.798175 10179 consensus_queue.cc:1059] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.050s
I20260503 13:58:34.811038 10547 consensus_queue.cc:1048] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
W20260503 13:58:34.914040 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 120790us
I20260503 13:58:34.847479 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:34.829779 10660 consensus_queue.cc:1059] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
I20260503 13:58:34.848284 10537 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:34.860632 10629 consensus_queue.cc:1048] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260503 13:58:34.867619 10347 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:34.860778 10372 consensus_queue.cc:1175] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "428b342dc4f041df956015e7d7cd173a" responder_term: 6 status { last_received { term: 4 index: 3 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]" } } last_received_current_leader { term: 0 index: 0 } }
I20260503 13:58:34.825537 10199 consensus_queue.cc:1059] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.047s
I20260503 13:58:34.998901 9448 raft_consensus.cc:1240] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:34.808120 10459 consensus_queue.cc:1059] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260503 13:58:34.836440 9522 raft_consensus.cc:2410] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 4 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260503 13:58:34.842095 10544 consensus_queue.cc:1059] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260503 13:58:34.812045 10639 consensus_queue.cc:1059] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260503 13:58:35.000993 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.081521 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:35.081938 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:35.082192 9601 raft_consensus.cc:2393] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 6: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:34.841759 10558 consensus_queue.cc:237] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:34.917513 9452 raft_consensus.cc:1240] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260503 13:58:34.808000 10649 consensus_queue.cc:1059] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260503 13:58:35.001834 10253 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:35.004040 9527 raft_consensus.cc:2410] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260503 13:58:35.076733 10170 consensus_queue.cc:260] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.103147 10170 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:35.105073 9599 raft_consensus.cc:1240] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:35.073079 10193 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.004362 10676 raft_consensus.cc:2749] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: Vote denied by peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997) with higher term. Message: Invalid argument: T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2.
I20260503 13:58:35.005924 10243 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:35.108969 9524 raft_consensus.cc:1240] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260503 13:58:35.006717 10655 raft_consensus.cc:515] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.007426 9444 raft_consensus.cc:1275] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:35.007831 10580 raft_consensus.cc:2749] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260503 13:58:35.008231 10703 log.cc:927] Time spent T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.163s user 0.001s sys 0.000s
I20260503 13:58:35.008348 10641 raft_consensus.cc:3055] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:35.120738 10641 raft_consensus.cc:740] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
W20260503 13:58:35.008535 10702 log.cc:927] Time spent T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.195s user 0.002s sys 0.000s
I20260503 13:58:35.119216 10336 consensus_queue.cc:1059] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.003s
I20260503 13:58:35.074014 10625 consensus_queue.cc:1059] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260503 13:58:35.122913 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 391) took 1959 ms. Trace:
I20260503 13:58:35.124171 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 482) took 1352 ms. Trace:
I20260503 13:58:35.099124 10544 consensus_queue.cc:1243] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer 428b342dc4f041df956015e7d7cd173a log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260503 13:58:35.010393 10283 consensus_queue.cc:260] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.011504 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:35.012449 9450 raft_consensus.cc:1275] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260503 13:58:35.013501 10647 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:35.008671 10698 log.cc:927] Time spent T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.242s user 0.002s sys 0.000s
W20260503 13:58:35.016366 10358 consensus_meta.cc:287] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.516s user 0.008s sys 0.000s
I20260503 13:58:35.016773 10587 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:35.018484 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:35.018865 9642 raft_consensus.cc:359] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.020952 9491 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:35.023108 9520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 469) took 2073 ms. Trace:
I20260503 13:58:35.026568 9416 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.027240 10652 consensus_queue.cc:1059] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:35.028379 9446 raft_consensus.cc:1275] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:34.813159 10630 consensus_queue.cc:1059] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260503 13:58:34.798632 10463 consensus_queue.cc:1059] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.049s
I20260503 13:58:35.030061 9644 raft_consensus.cc:359] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:35.036702 9521 rpcz_store.cc:269] 0503 13:58:33.508646 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.552873 (+ 44227us) service_pool.cc:224] Handling call
0503 13:58:34.840415 (+1287542us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54003fdc54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:34.840660 (+ 245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2595304192}
I20260503 13:58:35.059552 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:35.032594 9598 rpcz_store.cc:276] 0503 13:58:31.465228 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.328734 (+2863506us) service_pool.cc:224] Handling call
0503 13:58:34.680331 (+351597us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:34.696279 (+ 15948us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.839585 (+143306us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b54002bacd0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:34.839710 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":704530176}
I20260503 13:58:35.068701 9519 raft_consensus.cc:1240] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
W20260503 13:58:34.871868 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:35.069149 9525 raft_consensus.cc:1240] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:35.069317 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.069921 9523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 421) took 1084 ms. Trace:
I20260503 13:58:35.072242 9643 tablet_bootstrap.cc:492] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:35.092914 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.075417 10640 consensus_queue.cc:1059] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.026s
I20260503 13:58:35.124533 9528 rpcz_store.cc:276] 0503 13:58:33.771420 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.725587 (+954167us) service_pool.cc:224] Handling call
0503 13:58:34.727127 (+ 1540us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.775874 (+ 48747us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:35.124010 (+348136us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b54003d40d0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:35.124120 (+ 110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2903296}
I20260503 13:58:35.123238 9597 rpcz_store.cc:276] 0503 13:58:33.163158 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.607158 (+1444000us) service_pool.cc:224] Handling call
0503 13:58:34.607407 (+ 249us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:34.613148 (+ 5741us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:34.613177 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:34.622411 (+ 9234us) log.cc:844] Serialized 26 byte log entry
0503 13:58:34.657183 (+ 34772us) raft_consensus.cc:1645] Marking committed up to 4
0503 13:58:34.675519 (+ 18336us) log.cc:844] Serialized 14 byte log entry
0503 13:58:34.676155 (+ 636us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:34.676222 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:35.122744 (+446522us) raft_consensus.cc:1685] finished
0503 13:58:35.122764 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:35.122864 (+ 100us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2279,"prepare.queue_time_us":19931,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":5811,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":2772,"thread_start_us":21340,"threads_started":1,"wal-append.queue_time_us":140221}
I20260503 13:58:35.099351 10639 raft_consensus.cc:3055] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:35.127780 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.189059 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:35.189347 9642 raft_consensus.cc:385] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260503 13:58:35.189652 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 168762us
I20260503 13:58:35.189690 9642 raft_consensus.cc:740] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:35.190151 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.190637 9642 consensus_queue.cc:260] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.191460 9491 leader_election.cc:304] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:35.192530 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.202157 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.202698 9601 raft_consensus.cc:3060] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:35.205351 9644 raft_consensus.cc:385] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:35.205647 9644 raft_consensus.cc:740] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:35.206336 9644 consensus_queue.cc:260] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.188688 9523 rpcz_store.cc:276] 0503 13:58:33.985792 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.794783 (+808991us) service_pool.cc:224] Handling call
0503 13:58:34.840151 (+ 45368us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.840328 (+ 177us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:34.861697 (+ 21369us) log.cc:844] Serialized 14 byte log entry
0503 13:58:35.069716 (+208019us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:35.069757 (+ 41us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:35.069781 (+ 24us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:35.069821 (+ 40us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:35.069884 (+ 63us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":21015,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":1470,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":4151,"spinlock_wait_cycles":1615104,"thread_start_us":67572,"threads_started":1,"wal-append.queue_time_us":82039}
I20260503 13:58:35.117853 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.209556 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 432) took 1076 ms. Trace:
I20260503 13:58:35.211002 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.211652 9602 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:35.207104 9520 rpcz_store.cc:276] 0503 13:58:32.950427 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:33.969726 (+1019299us) service_pool.cc:224] Handling call
0503 13:58:34.669602 (+699876us) spinlock_profiling.cc:243] Waited 8.3 ms on lock 0x7b5800004598. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:34.726604 (+ 57002us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:34.776208 (+ 49604us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:35.022917 (+246709us) spinlock_profiling.cc:243] Waited 56.8 ms on lock 0x7b54003ff050. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:35.023054 (+ 137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149799680}
I20260503 13:58:35.211177 9525 rpcz_store.cc:276] 0503 13:58:34.133270 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.064444 (+931174us) service_pool.cc:224] Handling call
0503 13:58:35.069004 (+ 4560us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:35.209330 (+140326us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:35.209506 (+ 176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:35.213918 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:35.216631 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 190125us
I20260503 13:58:35.217546 9416 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.220206 9525 raft_consensus.cc:1275] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
W20260503 13:58:35.216547 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:35.221639 9527 raft_consensus.cc:1240] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:35.220541 9525 pending_rounds.cc:85] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a: Aborting all ops after (but not including) 1
I20260503 13:58:35.124825 10564 raft_consensus.cc:3055] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 6 LEADER]: Stepping down as leader of term 6
W20260503 13:58:35.224186 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 115341us
I20260503 13:58:35.093545 10635 consensus_queue.cc:260] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.223289 9525 pending_rounds.cc:107] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260503 13:58:35.100414 10253 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.225585 9524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 415) took 1241 ms. Trace:
I20260503 13:58:35.186956 10649 consensus_queue.cc:1048] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.093s
I20260503 13:58:35.223031 9596 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:35.230654 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.118767 10706 consensus_queue.cc:1059] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.104s
I20260503 13:58:35.245199 9519 raft_consensus.cc:1275] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:35.247637 9520 raft_consensus.cc:1240] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:35.261112 9644 ts_tablet_manager.cc:1434] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.561s user 0.006s sys 0.008s
I20260503 13:58:35.211638 10336 raft_consensus.cc:2804] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260503 13:58:35.217386 9594 consensus_meta.cc:287] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.700s user 0.006s sys 0.010s
I20260503 13:58:35.225296 10283 raft_consensus.cc:3060] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
W20260503 13:58:35.289597 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 56743.1us
I20260503 13:58:35.288919 9521 raft_consensus.cc:1275] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260503 13:58:35.289551 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 185456us
I20260503 13:58:35.290735 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.226620 10193 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:35.248657 9526 raft_consensus.cc:1275] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:35.268175 9525 raft_consensus.cc:2889] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:35.197337 10639 raft_consensus.cc:740] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:35.270295 10635 raft_consensus.cc:3060] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:35.221165 10544 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:35.271142 9524 rpcz_store.cc:276] 0503 13:58:33.984177 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.726256 (+742079us) service_pool.cc:224] Handling call
0503 13:58:34.726541 (+ 285us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:35.109240 (+382699us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:35.225350 (+116110us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b540038f254. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:35.225531 (+ 181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":789444992}
I20260503 13:58:35.271083 9601 raft_consensus.cc:2468] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:35.313611 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:35.314352 9601 raft_consensus.cc:2393] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:35.274747 9643 tablet_bootstrap.cc:654] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:35.276304 10253 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:35.131300 10631 consensus_queue.cc:1059] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.104s
I20260503 13:58:35.277304 10347 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:35.278026 10655 leader_election.cc:290] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:35.279112 10295 raft_consensus.cc:515] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.192404 10676 consensus_queue.cc:1048] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.075s
I20260503 13:58:35.280766 10630 raft_consensus.cc:3055] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:35.231648 9602 raft_consensus.cc:2468] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
W20260503 13:58:35.284632 9644 ts_tablet_manager.cc:1467] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6: Tablet startup took 584ms
I20260503 13:58:35.285146 9642 ts_tablet_manager.cc:1434] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.574s user 0.011s sys 0.011s
W20260503 13:58:35.281558 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 90169.8us
W20260503 13:58:35.332731 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 111354us
I20260503 13:58:35.290769 10634 consensus_queue.cc:1048] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.037s
I20260503 13:58:35.316895 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:35.232077 10564 raft_consensus.cc:740] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:35.224858 9595 raft_consensus.cc:2376] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4. Re-sending same reply.
W20260503 13:58:35.335578 9642 ts_tablet_manager.cc:1467] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a: Tablet startup took 624ms
I20260503 13:58:35.124758 10486 consensus_queue.cc:237] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.298125 10640 raft_consensus.cc:2749] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260503 13:58:35.298269 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:35.125797 10542 consensus_queue.cc:237] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.291642 9567 leader_election.cc:304] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:35.124756 10641 consensus_queue.cc:260] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.224184 10587 raft_consensus.cc:2749] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:35.302165 9523 raft_consensus.cc:1240] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 6 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260503 13:58:35.317212 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.289179 10673 raft_consensus.cc:2804] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:35.285691 10336 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:35.301358 10713 raft_consensus.cc:3055] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:35.281389 10714 consensus_queue.cc:1059] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.086s
I20260503 13:58:35.315822 10463 raft_consensus.cc:3055] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:35.283603 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.303145 10580 consensus_queue.cc:1059] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
W20260503 13:58:35.334103 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 116644us
I20260503 13:58:35.336035 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:35.396174 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:35.337262 9491 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.340164 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.317271 10544 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:35.335237 9644 ts_tablet_manager.cc:1469] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6: Trace:
0503 13:58:05.617432 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.310782 (+693350us) service_pool.cc:224] Handling call
0503 13:58:06.313568 (+ 2786us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.313649 (+ 81us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.403739 (+ 90090us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.435299 (+ 31560us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.439557 (+ 4258us) inbound_call.cc:177] Queueing success response
0503 13:58:34.200671 (+27761114us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0503 13:58:34.700391 (+499720us) ts_tablet_manager.cc:1435] Starting tablet replica
0503 13:58:35.024795 (+324404us) tablet_replica.cc:261] Starting instrumentation
0503 13:58:35.028672 (+ 3877us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":17368,"raft.queue_time_us":16652,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":7888,"tablet-open.queue_time_us":27761190}
W20260503 13:58:35.335857 9642 ts_tablet_manager.cc:1469] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a: Trace:
0503 13:58:05.622835 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.025208 (+402373us) service_pool.cc:224] Handling call
0503 13:58:06.026853 (+ 1645us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.026938 (+ 85us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.146469 (+119531us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.223854 (+ 77385us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.224241 (+ 387us) inbound_call.cc:177] Queueing success response
0503 13:58:34.076886 (+27852645us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0503 13:58:34.711327 (+634441us) ts_tablet_manager.cc:1435] Starting tablet replica
0503 13:58:34.999989 (+288662us) tablet_replica.cc:261] Starting instrumentation
0503 13:58:35.000464 (+ 475us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":3235,"raft.queue_time_us":38452,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":47752,"tablet-open.queue_time_us":27852720}
W20260503 13:58:35.342008 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 50456.4us
I20260503 13:58:35.343953 9601 raft_consensus.cc:2393] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:35.346246 10673 raft_consensus.cc:697] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:35.342149 10372 raft_consensus.cc:493] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:35.335350 10253 consensus_queue.cc:1059] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260503 13:58:35.397826 9598 raft_consensus.cc:1275] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:35.346628 10336 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:35.409379 9598 raft_consensus.cc:1275] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:35.400007 9416 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.344729 10459 raft_consensus.cc:2749] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:35.401754 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 392) took 2238 ms. Trace:
I20260503 13:58:35.402249 9446 raft_consensus.cc:2393] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
W20260503 13:58:35.403083 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 65884.1us
I20260503 13:58:35.419205 9491 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:35.420266 9491 leader_election.cc:343] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
W20260503 13:58:35.421126 9491 leader_election.cc:343] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:35.421473 9491 leader_election.cc:304] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.332419 10630 raft_consensus.cc:740] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:35.397210 10463 raft_consensus.cc:740] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:35.396458 9602 raft_consensus.cc:2393] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 6: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:35.405277 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:35.346408 10647 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:35.406972 9449 raft_consensus.cc:1275] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260503 13:58:35.429869 9598 raft_consensus.cc:1240] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 3. Ops: [1.1-1.1]
I20260503 13:58:35.429922 9602 raft_consensus.cc:3060] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:58:35.407404 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:35.410979 9445 raft_consensus.cc:1240] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:35.346268 10641 raft_consensus.cc:3060] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:35.412416 9599 rpcz_store.cc:276] 0503 13:58:33.163403 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.608154 (+1444751us) service_pool.cc:224] Handling call
0503 13:58:34.608394 (+ 240us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:35.105480 (+497086us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:35.401554 (+296074us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b54002611d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:35.401701 (+ 147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1141862144}
I20260503 13:58:35.409126 10633 raft_consensus.cc:2749] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260503 13:58:35.433727 9599 raft_consensus.cc:1240] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:35.401630 9567 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:35.436184 9567 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:35.414747 9416 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:35.340198 10193 consensus_queue.cc:1059] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.062s
I20260503 13:58:35.341272 10564 consensus_queue.cc:260] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.463923 9599 raft_consensus.cc:1240] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:35.422444 9491 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.425963 9642 tablet_bootstrap.cc:492] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:35.423480 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:35.411504 10698 log.cc:927] Time spent T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260503 13:58:35.465370 9599 raft_consensus.cc:1275] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260503 13:58:35.467259 9602 raft_consensus.cc:1240] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: []
W20260503 13:58:35.438529 9416 leader_election.cc:343] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:35.468101 9416 leader_election.cc:304] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.346721 10713 raft_consensus.cc:740] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:35.335029 10639 consensus_queue.cc:260] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.429438 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:35.470395 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 187553us
I20260503 13:58:35.471894 9599 raft_consensus.cc:1275] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:35.472975 9644 tablet_bootstrap.cc:492] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
I20260503 13:58:35.463063 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 495) took 1154 ms. Trace:
I20260503 13:58:35.429716 10543 raft_consensus.cc:3055] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:35.416000 10640 consensus_queue.cc:1059] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
W20260503 13:58:35.476547 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 54162.6us
I20260503 13:58:35.433740 10676 raft_consensus.cc:2749] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260503 13:58:35.489657 9599 raft_consensus.cc:1275] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:35.433374 10347 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:35.489939 9491 leader_election.cc:343] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:35.490360 9491 leader_election.cc:304] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.410979 10372 raft_consensus.cc:515] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.477140 9528 rpcz_store.cc:276] 0503 13:58:34.308968 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.191263 (+882295us) service_pool.cc:224] Handling call
0503 13:58:35.191516 (+ 253us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:35.191708 (+ 192us) raft_consensus.cc:1534] Early marking committed up to index 4
0503 13:58:35.224846 (+ 33138us) log.cc:844] Serialized 14 byte log entry
0503 13:58:35.432160 (+207314us) log.cc:844] Serialized 14 byte log entry
0503 13:58:35.462635 (+ 30475us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:35.462705 (+ 70us) raft_consensus.cc:1645] Marking committed up to 4
0503 13:58:35.462752 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:35.462837 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:35.462990 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3160,"prepare.queue_time_us":31361,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"raft.queue_time_us":28472,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":2559,"thread_start_us":205877,"threads_started":2,"wal-append.queue_time_us":117050}
I20260503 13:58:35.490829 9599 raft_consensus.cc:1275] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260503 13:58:35.492010 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 53512.7us
I20260503 13:58:35.492712 9416 leader_election.cc:304] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:35.494695 10652 raft_consensus.cc:2749] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:35.495297 10652 raft_consensus.cc:2804] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:35.495695 10652 raft_consensus.cc:697] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
W20260503 13:58:35.499593 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 162653us
I20260503 13:58:35.424429 10721 consensus_queue.cc:1048] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.053s
I20260503 13:58:35.469463 10647 raft_consensus.cc:2749] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260503 13:58:35.464314 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:35.426815 10719 consensus_queue.cc:1059] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.075s
I20260503 13:58:35.516901 9599 raft_consensus.cc:1240] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260503 13:58:35.520625 9491 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.521417 9599 raft_consensus.cc:1240] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260503 13:58:35.522657 9599 raft_consensus.cc:1240] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 6 LEADER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260503 13:58:35.477021 10543 raft_consensus.cc:740] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:35.498608 10535 raft_consensus.cc:2749] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260503 13:58:35.539233 9491 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
W20260503 13:58:35.541491 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 104210us
I20260503 13:58:35.475112 10722 raft_consensus.cc:2749] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260503 13:58:35.500108 10564 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:35.527478 10639 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:35.543293 9491 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:35.538030 10717 log.cc:927] Time spent T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.217s user 0.001s sys 0.000s
I20260503 13:58:35.500268 10723 raft_consensus.cc:2804] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:35.556645 9491 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.524780 9593 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 6.
I20260503 13:58:35.499362 10347 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:35.499307 10649 consensus_queue.cc:1175] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4bea375a55034750a1f8804b0db97868" responder_term: 6 status { last_received { term: 3 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260503 13:58:35.559365 9491 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:35.537810 10726 raft_consensus.cc:2749] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:35.540424 10295 leader_election.cc:290] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:35.542536 10647 raft_consensus.cc:2749] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260503 13:58:35.498162 10630 consensus_queue.cc:260] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:35.543359 9566 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:35.496507 10652 consensus_queue.cc:237] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.552714 10253 raft_consensus.cc:2749] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260503 13:58:35.552387 10646 raft_consensus.cc:2804] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:35.499359 10463 consensus_queue.cc:260] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.558475 10535 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:35.575291 9491 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.516647 10673 consensus_queue.cc:237] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:35.578833 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:35.565138 10676 raft_consensus.cc:2749] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260503 13:58:35.577783 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 85137.8us
I20260503 13:58:35.582825 9416 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.615077 9416 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.618661 9644 tablet_bootstrap.cc:654] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:35.568336 10723 raft_consensus.cc:697] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:35.626842 9450 raft_consensus.cc:1240] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:35.573072 10558 raft_consensus.cc:2804] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:35.633400 9416 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.633483 10723 consensus_queue.cc:237] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.572952 10735 raft_consensus.cc:2749] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260503 13:58:35.586553 10726 raft_consensus.cc:2749] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:35.582371 10646 raft_consensus.cc:697] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:35.582798 9595 raft_consensus.cc:1240] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:35.571362 10722 consensus_queue.cc:1048] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.061s
W20260503 13:58:35.690274 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 119142us
W20260503 13:58:35.626740 10718 log.cc:927] Time spent T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.224s user 0.000s sys 0.001s
I20260503 13:58:35.602540 10544 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:35.605118 10630 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:35.580452 10580 consensus_queue.cc:1059] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.074s
W20260503 13:58:35.650960 10720 log.cc:927] Time spent T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.243s user 0.001s sys 0.000s
I20260503 13:58:35.554776 10713 consensus_queue.cc:260] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.573475 10543 consensus_queue.cc:260] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.664032 10731 consensus_queue.cc:1059] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.054s
I20260503 13:58:35.656816 10719 raft_consensus.cc:2749] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:35.657922 9593 raft_consensus.cc:3055] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:35.601627 10729 consensus_queue.cc:1048] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.133s
I20260503 13:58:35.664527 10721 raft_consensus.cc:2749] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260503 13:58:35.661731 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:35.663280 10558 raft_consensus.cc:697] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:35.739733 9642 tablet_bootstrap.cc:654] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:35.666112 10676 consensus_queue.cc:1059] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
I20260503 13:58:35.583305 10724 consensus_queue.cc:1048] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.130s
W20260503 13:58:35.733124 10711 log.cc:927] Time spent T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260503 13:58:35.672652 10723 consensus_queue.cc:1059] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260503 13:58:35.729449 10726 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:35.735399 10735 raft_consensus.cc:493] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:35.740442 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 107120us
W20260503 13:58:35.699146 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 157658us
I20260503 13:58:35.743713 9593 raft_consensus.cc:740] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:35.749575 10372 leader_election.cc:290] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:35.756963 10725 log.cc:927] Time spent T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.229s user 0.000s sys 0.001s
I20260503 13:58:35.729421 10732 consensus_queue.cc:1048] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260503 13:58:35.768946 9416 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:35.739841 10633 consensus_queue.cc:1059] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.021s
W20260503 13:58:35.773222 9418 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:35.757125 10740 raft_consensus.cc:2749] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260503 13:58:35.776398 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:35.794168 9416 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:35.792443 10558 consensus_queue.cc:237] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.738313 10734 consensus_queue.cc:1059] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260503 13:58:35.771481 10726 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:35.807799 10727 log.cc:927] Time spent T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.231s user 0.001s sys 0.000s
I20260503 13:58:35.776481 10347 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:35.812183 9601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 405) took 1185 ms. Trace:
I20260503 13:58:35.801514 10736 raft_consensus.cc:493] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:35.812520 9601 rpcz_store.cc:276] 0503 13:58:34.626466 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.398878 (+772412us) service_pool.cc:224] Handling call
0503 13:58:35.399159 (+ 281us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:35.555573 (+156414us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:35.555602 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:35.567614 (+ 12012us) log.cc:844] Serialized 26 byte log entry
0503 13:58:35.620775 (+ 53161us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:35.621055 (+ 280us) log.cc:844] Serialized 14 byte log entry
0503 13:58:35.730400 (+109345us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:35.730796 (+ 396us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:35.811999 (+ 81203us) raft_consensus.cc:1685] finished
0503 13:58:35.812017 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:35.812132 (+ 115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50063,"prepare.queue_time_us":152,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":280600,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":39514,"thread_start_us":233591,"threads_started":3,"wal-append.queue_time_us":180795}
I20260503 13:58:35.819161 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.759388 10646 consensus_queue.cc:237] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.806257 9595 raft_consensus.cc:1275] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260503 13:58:35.823472 10537 consensus_meta.cc:287] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.749s user 0.009s sys 0.000s
I20260503 13:58:35.810534 10743 raft_consensus.cc:2749] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260503 13:58:35.817800 10347 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:35.771255 10735 raft_consensus.cc:515] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.815627 10722 raft_consensus.cc:2804] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:35.818389 10676 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:35.818018 10724 raft_consensus.cc:493] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:35.828989 10734 consensus_queue.cc:1243] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer 428b342dc4f041df956015e7d7cd173a log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260503 13:58:35.840915 10726 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:35.841128 10733 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:35.841308 10722 raft_consensus.cc:697] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:35.810604 9593 consensus_queue.cc:260] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.821684 10736 raft_consensus.cc:515] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.837872 10730 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:35.820492 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:35.837611 10676 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
@ 0x7f9f3b33d7f1 abort at ??:0
@ 0x2d3616 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
I20260503 13:58:35.859735 10676 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:35.853155 10756 raft_consensus.cc:3055] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:35.862726 10756 raft_consensus.cc:740] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:35.863432 10756 consensus_queue.cc:260] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.865157 10722 consensus_queue.cc:237] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:35.867511 10635 consensus_meta.cc:287] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.555s user 0.005s sys 0.004s
I20260503 13:58:35.836398 10347 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.870186 9644 tablet_bootstrap.cc:492] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
I20260503 13:58:35.881428 9644 ts_tablet_manager.cc:1403] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 0.456s user 0.022s sys 0.004s
I20260503 13:58:35.838903 10724 raft_consensus.cc:515] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.883885 10736 leader_election.cc:290] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:35.884837 9644 raft_consensus.cc:359] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.885442 9644 raft_consensus.cc:385] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:35.885684 9644 raft_consensus.cc:740] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:35.886174 9644 consensus_queue.cc:260] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.855801 10733 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.893878 10347 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:35.894970 10733 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:35.896981 10243 consensus_meta.cc:287] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.788s user 0.012s sys 0.001s
I20260503 13:58:35.904495 9644 ts_tablet_manager.cc:1434] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.023s user 0.004s sys 0.002s
W20260503 13:58:35.908458 10283 consensus_meta.cc:287] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.612s user 0.002s sys 0.007s
I20260503 13:58:35.909336 9596 raft_consensus.cc:2410] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260503 13:58:35.886759 10735 leader_election.cc:290] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:35.911132 9596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 425) took 1213 ms. Trace:
I20260503 13:58:35.914896 9596 rpcz_store.cc:276] 0503 13:58:34.697942 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.283209 (+585267us) service_pool.cc:224] Handling call
0503 13:58:35.910955 (+627746us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b54002bb6d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:35.911078 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1181388544}
I20260503 13:58:35.913359 9597 raft_consensus.cc:2468] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:35.883937 10724 leader_election.cc:290] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:35.917100 10535 raft_consensus.cc:2749] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:35.977206 9644 tablet_bootstrap.cc:492] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6: Bootstrap starting.
W20260503 13:58:35.941051 10336 consensus_meta.cc:287] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.513s user 0.007s sys 0.001s
I20260503 13:58:35.978493 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:35.978695 9524 raft_consensus.cc:1275] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:35.979672 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:35.940521 9522 raft_consensus.cc:1240] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 6 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:35.984867 9523 raft_consensus.cc:1240] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [term 6 LEADER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260503 13:58:35.988274 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:35.988883 9520 raft_consensus.cc:2468] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:35.993196 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:35.993748 9526 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:35.994251 9522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 491) took 1762 ms. Trace:
I20260503 13:58:35.861929 10730 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:35.996209 9522 rpcz_store.cc:276] 0503 13:58:34.232229 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.098441 (+866212us) service_pool.cc:224] Handling call
0503 13:58:35.185059 (+ 86618us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:35.940348 (+755289us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54002425a0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41aa31a0 00007f9f41b2d5a0 00007f9f41b2e611 00007f9f41b2b16f 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c
0503 13:58:35.982936 (+ 42588us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:35.994193 (+ 11257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":285203328}
I20260503 13:58:35.981624 9452 raft_consensus.cc:1275] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:35.999439 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:35.999966 9524 raft_consensus.cc:3060] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:36.002125 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:36.006072 9523 raft_consensus.cc:1275] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260503 13:58:36.004726 9444 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.013s sys 0.007s
I20260503 13:58:35.981920 10336 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.015277 9524 raft_consensus.cc:2468] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:36.030025 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 418) took 1387 ms. Trace:
W20260503 13:58:36.022719 9444 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s user 0.013s sys 0.007s
I20260503 13:58:36.032007 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:36.023759 10668 raft_consensus.cc:3060] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:36.038391 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:36.038568 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 456562us
I20260503 13:58:36.038983 9526 raft_consensus.cc:2468] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:36.029372 10283 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:36.026394 10511 raft_consensus.cc:3055] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:36.039884 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:36.043233 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 394) took 1419 ms. Trace:
I20260503 13:58:36.047987 9597 rpcz_store.cc:276] 0503 13:58:34.623961 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.213667 (+589706us) service_pool.cc:224] Handling call
0503 13:58:36.043091 (+829424us) spinlock_profiling.cc:243] Waited 683 ms on lock 0x7b54002ba2d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:36.043183 (+ 92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1571399424}
I20260503 13:58:36.047014 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:36.046402 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:36.049228 9449 raft_consensus.cc:2468] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
W20260503 13:58:36.038406 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:36.030581 9598 rpcz_store.cc:276] 0503 13:58:34.642636 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.432050 (+789414us) service_pool.cc:224] Handling call
0503 13:58:35.432299 (+ 249us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:35.432493 (+ 194us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:35.603646 (+171153us) log.cc:844] Serialized 14 byte log entry
0503 13:58:36.029722 (+426076us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:36.029770 (+ 48us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:36.029806 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:36.029875 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:36.029976 (+ 101us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":111364,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":1730,"raft.queue_time_us":70,"thread_start_us":559569,"threads_started":2}
I20260503 13:58:36.047766 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "754ac001a92e4dd6a0b6620803b83722" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:36.057032 9567 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:36.060034 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:36.059567 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 369) took 3583 ms. Trace:
I20260503 13:58:36.072412 9567 leader_election.cc:304] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:36.062754 9594 rpcz_store.cc:276] 0503 13:58:32.475651 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.484720 (+2009069us) service_pool.cc:224] Handling call
0503 13:58:34.516513 (+ 31793us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:35.320101 (+803588us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:35.321829 (+ 1728us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:35.345868 (+ 24039us) log.cc:844] Serialized 26 byte log entry
0503 13:58:35.715094 (+369226us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:35.737713 (+ 22619us) log.cc:844] Serialized 14 byte log entry
0503 13:58:36.059262 (+321549us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:36.059348 (+ 86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:36.059391 (+ 43us) raft_consensus.cc:1685] finished
0503 13:58:36.059406 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:36.059513 (+ 107us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43874,"prepare.queue_time_us":4549,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":94630,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":11064,"thread_start_us":491057,"threads_started":2,"wal-append.queue_time_us":408877}
I20260503 13:58:36.047640 10511 raft_consensus.cc:740] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:36.107226 10511 consensus_queue.cc:260] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.052002 10639 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:36.059553 10616 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:36.063060 10199 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:36.123296 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 50963.6us
I20260503 13:58:36.166116 9567 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:36.070118 9452 raft_consensus.cc:1275] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260503 13:58:36.162170 9446 raft_consensus.cc:2393] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 6: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:36.170301 9520 raft_consensus.cc:1240] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:36.070452 10654 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:36.111920 9525 raft_consensus.cc:1240] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
W20260503 13:58:36.174127 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:36.177598 9449 raft_consensus.cc:1275] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260503 13:58:36.049933 10283 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.188532 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d04573f959de406a9590a1603f633e7e" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:36.197201 9520 raft_consensus.cc:2468] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:36.198382 9520 raft_consensus.cc:1240] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:36.199424 9520 raft_consensus.cc:1240] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:36.200421 9520 raft_consensus.cc:1240] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:36.110644 10544 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:36.109302 10587 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:36.065663 10735 raft_consensus.cc:2804] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:36.065966 10676 consensus_queue.cc:1048] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.112s
I20260503 13:58:36.164927 10511 raft_consensus.cc:3060] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:36.168725 10347 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:36.178247 10733 raft_consensus.cc:2804] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election won for term 3
@ 0x7f9f3e111da4 google::LogMessage::SendToLog() at ??:0
I20260503 13:58:36.215467 9418 leader_election.cc:304] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:36.215669 9566 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:36.109925 10731 consensus_queue.cc:1048] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.098s
I20260503 13:58:36.105525 10726 consensus_queue.cc:1048] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.129s
I20260503 13:58:36.178025 10736 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:36.217792 10347 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.251309 10733 raft_consensus.cc:697] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
W20260503 13:58:36.333711 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 118298us
W20260503 13:58:36.278266 9554 tablet.cc:2404] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260503 13:58:36.339370 10618 raft_consensus.cc:3055] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:36.335381 10736 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.339727 10618 raft_consensus.cc:740] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:36.340814 10347 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:36.340492 10618 consensus_queue.cc:260] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.341555 10660 raft_consensus.cc:3055] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:36.341908 10660 raft_consensus.cc:740] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:36.342538 10660 consensus_queue.cc:260] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.343410 10660 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
W20260503 13:58:36.344698 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:36.345213 9643 tablet_bootstrap.cc:492] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:36.332800 10719 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:36.355194 9418 leader_election.cc:304] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:36.213529 10735 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260503 13:58:36.273402 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 479263us
I20260503 13:58:36.214430 10587 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.233444 10730 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:36.301442 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:36.209683 10653 raft_consensus.cc:493] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:36.243686 9444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 455) took 1116 ms. Trace:
I20260503 13:58:36.244866 10768 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:36.166353 10673 consensus_queue.cc:1048] T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.051s
I20260503 13:58:36.180506 10199 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.358776 10735 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:36.181066 10639 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.164777 10646 consensus_queue.cc:1048] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.053s
I20260503 13:58:36.202849 10724 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:36.214031 10564 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:36.233035 10372 consensus_queue.cc:1048] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260503 13:58:36.166584 10649 consensus_queue.cc:1048] T 1c480795fed14481acd9ca2d189b9583 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.051s
I20260503 13:58:36.162400 10547 consensus_queue.cc:1059] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.100s
I20260503 13:58:36.171546 10616 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.186861 10475 consensus_queue.cc:260] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.385532 10722 raft_consensus.cc:2804] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:36.241047 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "754ac001a92e4dd6a0b6620803b83722" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:36.214920 10544 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.400228 10722 raft_consensus.cc:493] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:36.400656 10722 raft_consensus.cc:3060] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:36.401069 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:36.401705 9452 raft_consensus.cc:2468] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:36.402506 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:36.404944 10756 raft_consensus.cc:3060] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:36.111285 10758 consensus_queue.cc:1059] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.162s
I20260503 13:58:36.319959 9453 raft_consensus.cc:1240] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:36.277709 10336 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:36.234350 10734 raft_consensus.cc:2804] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260503 13:58:36.233193 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 67136.9us
W20260503 13:58:36.420642 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 353090us
I20260503 13:58:36.329000 9566 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:36.209874 10654 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.305250 10766 raft_consensus.cc:493] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:36.331391 9602 raft_consensus.cc:1240] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:36.333281 9445 raft_consensus.cc:1240] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:36.362697 10741 raft_consensus.cc:2804] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:36.360339 10719 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.364439 9416 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:36.370870 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:36.386862 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 315485us
I20260503 13:58:36.388783 10767 raft_consensus.cc:493] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:36.360096 10743 consensus_queue.cc:1059] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.063s
I20260503 13:58:36.446955 10767 raft_consensus.cc:515] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.448881 10767 leader_election.cc:290] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:36.450013 10649 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:36.450441 10649 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.451706 10587 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:36.452111 10649 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:36.448953 9447 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.791s user 0.015s sys 0.029s
W20260503 13:58:36.452683 9447 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.795s user 0.015s sys 0.029s
I20260503 13:58:36.444399 10741 raft_consensus.cc:493] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:36.453500 10741 raft_consensus.cc:3060] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:36.423985 9492 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:36.390568 10283 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:36.363399 10771 consensus_queue.cc:1059] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260503 13:58:36.454106 9447 raft_consensus.cc:1240] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260503 13:58:36.436391 10734 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:36.442806 10730 raft_consensus.cc:3055] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:36.384717 10653 raft_consensus.cc:515] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.485513 9491 leader_election.cc:304] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:36.500793 9447 raft_consensus.cc:1275] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:36.505730 9447 raft_consensus.cc:1240] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260503 13:58:36.454407 10766 raft_consensus.cc:515] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.406461 10724 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.406554 10564 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.421005 10729 consensus_queue.cc:1059] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.080s
I20260503 13:58:36.386796 10768 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.454124 10733 consensus_queue.cc:237] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.534205 9524 raft_consensus.cc:1240] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [term 6 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260503 13:58:37.032369 9447 raft_consensus.cc:1275] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260503 13:58:37.030795 9643 ts_tablet_manager.cc:1403] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 1.994s user 0.050s sys 0.063s
W20260503 13:58:37.087904 9526 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.028s sys 0.027s
W20260503 13:58:37.038143 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 616119us
W20260503 13:58:37.116359 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 787382us
I20260503 13:58:36.513238 9642 tablet_bootstrap.cc:492] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:36.901607 9602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 428) took 2138 ms. Trace:
I20260503 13:58:37.123504 9524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 513) took 1089 ms. Trace:
I20260503 13:58:37.115453 9527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 439) took 2861 ms. Trace:
I20260503 13:58:36.917385 9445 raft_consensus.cc:1240] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 3. Current term is 4. Ops: []
W20260503 13:58:37.125939 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:36.502688 10777 log.cc:927] Time spent T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.158s user 0.001s sys 0.000s
W20260503 13:58:37.126535 9491 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.005s sys 0.001s
I20260503 13:58:36.728816 10561 raft_consensus.cc:3055] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:36.502512 10734 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:36.499402 10649 raft_consensus.cc:2804] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:37.130448 9527 rpcz_store.cc:276] 0503 13:58:34.254518 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.094952 (+840434us) service_pool.cc:224] Handling call
0503 13:58:35.095206 (+ 254us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:35.222756 (+127550us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:36.669119 (+1446363us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54003fe654. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:37.115374 (+446255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":290411264}
W20260503 13:58:37.124078 9526 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.668s user 0.029s sys 0.029s
I20260503 13:58:36.503227 10758 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:36.668462 10713 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:37.091624 10445 raft_consensus.cc:3055] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:37.059959 10785 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.124959 10744 raft_consensus.cc:2749] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260503 13:58:37.139811 9602 rpcz_store.cc:276] 0503 13:58:34.763167 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.474225 (+711058us) service_pool.cc:224] Handling call
0503 13:58:35.474472 (+ 247us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:36.441396 (+966924us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:36.901383 (+459987us) spinlock_profiling.cc:243] Waited 857 ms on lock 0x7b540031a6d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:36.901545 (+ 162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1970299776}
W20260503 13:58:37.173768 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:37.173691 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 43927.8us
W20260503 13:58:37.184803 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:37.185372 9566 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:36.810922 10736 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:36.907305 10616 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:36.533416 10654 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.128062 10587 raft_consensus.cc:493] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:37.153609 9567 leader_election.cc:343] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:36.693161 10673 raft_consensus.cc:2749] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:36.817117 10630 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:36.525655 10639 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:36.502303 10730 raft_consensus.cc:740] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
W20260503 13:58:36.895844 10763 log.cc:927] Time spent T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.603s user 0.003s sys 0.000s
I20260503 13:58:36.497885 10336 consensus_queue.cc:1048] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.649s
I20260503 13:58:36.806562 10638 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.128727 10544 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.131789 10771 consensus_queue.cc:1243] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer 428b342dc4f041df956015e7d7cd173a log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260503 13:58:37.180449 10372 consensus_queue.cc:1059] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260503 13:58:37.122699 10636 raft_consensus.cc:3055] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:37.152491 10543 raft_consensus.cc:3060] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:36.809859 10741 raft_consensus.cc:515] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:36.874985 10558 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:37.173202 10743 consensus_queue.cc:1243] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer 428b342dc4f041df956015e7d7cd173a log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260503 13:58:36.502418 10726 consensus_queue.cc:1048] T 8b9644d56a7f4556ae897658749f02e4 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.680s
I20260503 13:58:36.722692 10634 consensus_queue.cc:1048] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.459s
I20260503 13:58:36.530246 10722 raft_consensus.cc:515] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.174804 10561 raft_consensus.cc:740] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:37.180080 10649 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:37.188429 10785 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.204691 10638 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.190440 10744 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.185092 10653 leader_election.cc:290] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.178886 10769 consensus_queue.cc:1048] T 2c918e4ac84c4238a06220bc2817286f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260503 13:58:37.202764 10673 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:37.207019 10636 raft_consensus.cc:740] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:37.214903 10649 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:37.230859 10768 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.232064 10741 leader_election.cc:290] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.221344 10744 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.235030 10766 leader_election.cc:290] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.236111 10785 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:37.259093 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:37.264088 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:37.124619 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 463) took 1536 ms. Trace:
W20260503 13:58:37.147357 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 783330us
I20260503 13:58:37.123219 9447 pending_rounds.cc:85] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6: Aborting all ops after (but not including) 3
I20260503 13:58:37.265667 9453 rpcz_store.cc:276] 0503 13:58:35.588025 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.588124 (+ 99us) service_pool.cc:224] Handling call
0503 13:58:35.588367 (+ 243us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:36.405962 (+817595us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:36.891563 (+485601us) spinlock_profiling.cc:243] Waited 731 ms on lock 0x7b5400294ad4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:37.124523 (+232960us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1682056064}
I20260503 13:58:37.266990 9447 pending_rounds.cc:107] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260503 13:58:37.267171 9416 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:37.268253 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.268407 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:36.383149 9444 rpcz_store.cc:276] 0503 13:58:35.126985 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.127340 (+ 355us) service_pool.cc:224] Handling call
0503 13:58:35.229650 (+102310us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:35.428894 (+199244us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:35.431485 (+ 2591us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:35.488552 (+ 57067us) log.cc:844] Serialized 26 byte log entry
0503 13:58:36.067613 (+579061us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:36.084778 (+ 17165us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:36.108647 (+ 23869us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:36.243451 (+134804us) raft_consensus.cc:1685] finished
0503 13:58:36.243491 (+ 40us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:36.243630 (+ 139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34772,"raft.queue_time_us":113579,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":11456,"thread_start_us":708351,"threads_started":2,"wal-append.queue_time_us":698654}
I20260503 13:58:37.270061 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:37.270205 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.270972 9453 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:37.271445 9444 raft_consensus.cc:1275] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:37.123492 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:37.266069 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 499) took 1070 ms. Trace:
I20260503 13:58:37.273627 9452 raft_consensus.cc:3060] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:37.268674 9447 raft_consensus.cc:2889] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
@ 0x7f9f3e112950 google::LogMessage::Flush() at ??:0
I20260503 13:58:37.275089 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 500) took 1079 ms. Trace:
W20260503 13:58:37.173997 9491 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.669s user 0.006s sys 0.001s
I20260503 13:58:37.130328 9524 rpcz_store.cc:276] 0503 13:58:36.034122 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:36.041717 (+ 7595us) service_pool.cc:224] Handling call
0503 13:58:36.071625 (+ 29908us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:36.532453 (+460828us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b54002899a0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41aa31a0 00007f9f41b2d5a0 00007f9f41b2e611 00007f9f41b2b16f 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c
0503 13:58:37.123112 (+590659us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:37.123447 (+ 335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":567974784}
I20260503 13:58:37.277374 9519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 463) took 1301 ms. Trace:
I20260503 13:58:37.174059 9528 raft_consensus.cc:1240] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:37.277758 9519 rpcz_store.cc:276] 0503 13:58:35.975323 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.979721 (+ 4398us) service_pool.cc:224] Handling call
0503 13:58:35.980006 (+ 285us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:35.982188 (+ 2182us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:35.982220 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:35.987386 (+ 5166us) log.cc:844] Serialized 52 byte log entry
0503 13:58:36.473315 (+485929us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:36.473355 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:36.473421 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:37.276810 (+803389us) raft_consensus.cc:1685] finished
0503 13:58:37.276849 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:37.277126 (+ 277us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1812,"raft.queue_time_us":2393,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":7661,"thread_start_us":467910,"threads_started":1,"wal-append.queue_time_us":1190029}
I20260503 13:58:37.280264 9522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 464) took 1111 ms. Trace:
I20260503 13:58:37.280684 9522 rpcz_store.cc:276] 0503 13:58:36.169212 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:36.169343 (+ 131us) service_pool.cc:224] Handling call
0503 13:58:36.210104 (+ 40761us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:36.214705 (+ 4601us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:36.214731 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:36.273940 (+ 59209us) log.cc:844] Serialized 52 byte log entry
0503 13:58:36.721760 (+447820us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:36.721801 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:36.721876 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:37.279788 (+557912us) raft_consensus.cc:1685] finished
0503 13:58:37.279825 (+ 37us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:37.280093 (+ 268us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":34548,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"thread_start_us":400589,"threads_started":1,"wal-append.queue_time_us":898825}
I20260503 13:58:37.275450 9447 rpcz_store.cc:276] 0503 13:58:36.195330 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:37.031724 (+836394us) service_pool.cc:224] Handling call
0503 13:58:37.032025 (+ 301us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:37.274755 (+242730us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:37.275029 (+ 274us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:37.276319 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 771699us
I20260503 13:58:37.284106 9491 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:37.178486 9565 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:37.139420 9642 ts_tablet_manager.cc:1403] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 1.725s user 0.032s sys 0.036s
I20260503 13:58:37.273523 9445 rpcz_store.cc:276] 0503 13:58:36.195126 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:36.900139 (+705013us) service_pool.cc:224] Handling call
0503 13:58:36.900403 (+ 264us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:37.265301 (+364898us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:37.265803 (+ 502us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:37.125572 9644 tablet_bootstrap.cc:654] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:36.523743 10199 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.173501 10283 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.317504 10753 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:37.185706 10445 raft_consensus.cc:740] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:37.318015 10542 raft_consensus.cc:2749] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:37.318302 10753 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.319260 10445 consensus_queue.cc:260] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.188621 10506 consensus_queue.cc:1059] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 1, Time since last communication: 0.113s
I20260503 13:58:37.285722 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 465) took 1116 ms. Trace:
I20260503 13:58:37.296465 9642 raft_consensus.cc:359] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.322610 9642 raft_consensus.cc:385] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:37.322984 9642 raft_consensus.cc:740] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:37.321445 9526 rpcz_store.cc:276] 0503 13:58:36.169498 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:36.169627 (+ 129us) service_pool.cc:224] Handling call
0503 13:58:36.213817 (+ 44190us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:36.214967 (+ 1150us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:36.214994 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:36.218699 (+ 3705us) log.cc:844] Serialized 52 byte log entry
0503 13:58:36.424350 (+205651us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:37.277669 (+853319us) log.cc:844] Serialized 14 byte log entry
0503 13:58:37.285042 (+ 7373us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:37.285146 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:37.285235 (+ 89us) raft_consensus.cc:1685] finished
0503 13:58:37.285249 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:37.285519 (+ 270us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":821,"prepare.queue_time_us":788905,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":1807,"raft.queue_time_us":31070,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":65,"thread_start_us":909912,"threads_started":2,"wal-append.queue_time_us":944237}
I20260503 13:58:37.320477 10445 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:37.323782 9642 consensus_queue.cc:260] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.284518 9492 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:37.292915 9447 raft_consensus.cc:3055] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Stepping down as leader of term 1
W20260503 13:58:37.292707 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 105850us
I20260503 13:58:37.290021 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.327138 9447 raft_consensus.cc:740] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:37.299855 9643 raft_consensus.cc:359] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.329027 9643 raft_consensus.cc:385] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:37.329325 9643 raft_consensus.cc:740] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
W20260503 13:58:37.294574 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:37.292048 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 106558us
I20260503 13:58:37.293833 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:37.296968 9491 connection.cc:254] RPC call timeout handler was delayed by -1.0326917699999854s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19330000000000003s for a response
W20260503 13:58:37.285531 9416 leader_election.cc:343] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:37.333287 9416 leader_election.cc:304] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:37.188099 10758 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.334578 10631 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.369102 10631 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.370057 10475 raft_consensus.cc:3060] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:37.371620 10631 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.334815 10724 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.201850 10630 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.335642 10755 raft_consensus.cc:493] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.373576 10463 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:37.374141 10630 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.374707 10630 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.375322 10630 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.317880 10283 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.335557 10753 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.377207 10630 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.374142 10758 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.378026 10534 raft_consensus.cc:493] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.378073 10283 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.378691 10534 raft_consensus.cc:515] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.211669 10558 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.381127 10534 leader_election.cc:290] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.336009 10638 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:37.336849 10735 consensus_meta.cc:287] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.814s user 0.007s sys 0.000s
I20260503 13:58:37.337327 10647 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:37.220677 10730 consensus_queue.cc:260] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.343171 9642 ts_tablet_manager.cc:1434] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.051s user 0.010s sys 0.000s
I20260503 13:58:37.343791 10794 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.345359 10722 leader_election.cc:290] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:37.348083 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 310185us
I20260503 13:58:37.350080 10564 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.223966 10673 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.353242 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 456) took 1379 ms. Trace:
I20260503 13:58:37.354621 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 459) took 1380 ms. Trace:
I20260503 13:58:37.356792 10795 raft_consensus.cc:2749] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:37.358744 9593 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
W20260503 13:58:37.359509 9566 leader_election.cc:343] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:37.360049 10654 raft_consensus.cc:2749] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260503 13:58:37.360543 10743 raft_consensus.cc:2804] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:37.361449 9452 raft_consensus.cc:2410] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260503 13:58:37.361230 10796 consensus_queue.cc:1048] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260503 13:58:37.189399 10587 raft_consensus.cc:515] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.363385 9445 raft_consensus.cc:2468] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:37.364382 10336 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:37.402540 9528 rpcz_store.cc:276] 0503 13:58:35.973515 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.973637 (+ 122us) service_pool.cc:224] Handling call
0503 13:58:35.973931 (+ 294us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:37.278140 (+1304209us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:37.352834 (+ 74694us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54002071d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:37.353062 (+ 228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2759624448}
I20260503 13:58:37.367699 9447 consensus_queue.cc:260] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.384188 10755 raft_consensus.cc:515] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.381175 10558 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.407739 9566 leader_election.cc:304] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:37.385742 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:37.386147 9643 consensus_queue.cc:260] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.386029 10668 raft_consensus.cc:3055] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:37.389446 9453 raft_consensus.cc:2393] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:37.389948 10735 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.390336 10721 raft_consensus.cc:3055] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:37.390954 10647 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:37.392700 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 107324us
I20260503 13:58:37.393388 10653 raft_consensus.cc:2749] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260503 13:58:37.402819 9525 rpcz_store.cc:276] 0503 13:58:35.974312 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.980267 (+ 5955us) service_pool.cc:224] Handling call
0503 13:58:35.980500 (+ 233us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:36.188729 (+208229us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:37.354141 (+1165412us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b540027bad4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:37.354434 (+ 293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":301840640}
I20260503 13:58:37.317904 10726 consensus_queue.cc:1059] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
W20260503 13:58:37.403836 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 119664us
I20260503 13:58:37.401173 10794 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.421087 10726 consensus_queue.cc:1243] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer 1cb63886040047559d3893923cc9aaf6 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260503 13:58:37.422695 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 423) took 2159 ms. Trace:
I20260503 13:58:37.422827 10735 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.411194 10668 raft_consensus.cc:740] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:37.424594 10636 consensus_queue.cc:260] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.426195 10587 leader_election.cc:290] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.426443 10636 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:37.427079 9642 tablet_bootstrap.cc:492] T 754ac001a92e4dd6a0b6620803b83722 P 428b342dc4f041df956015e7d7cd173a: Bootstrap starting.
I20260503 13:58:37.423076 10459 raft_consensus.cc:3055] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:37.429618 10459 raft_consensus.cc:740] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:37.429396 10336 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.429908 10673 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.430678 10459 consensus_queue.cc:260] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.431653 10336 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.432508 10798 consensus_queue.cc:1059] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260503 13:58:37.433019 10744 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.424724 10730 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:37.438191 10647 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.425963 10668 consensus_queue.cc:260] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.425901 9447 raft_consensus.cc:3060] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
W20260503 13:58:37.436141 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:37.435159 10459 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:37.437716 10653 consensus_queue.cc:1048] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:58:37.444741 9452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 494) took 1242 ms. Trace:
I20260503 13:58:37.423061 9599 rpcz_store.cc:276] 0503 13:58:35.262801 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.523258 (+260457us) service_pool.cc:224] Handling call
0503 13:58:35.523528 (+ 270us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:35.627194 (+103666us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:35.627228 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:35.627640 (+ 412us) log.cc:844] Serialized 26 byte log entry
0503 13:58:35.726971 (+ 99331us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:35.748142 (+ 21171us) log.cc:844] Serialized 14 byte log entry
0503 13:58:35.801634 (+ 53492us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:37.422027 (+1620393us) spinlock_profiling.cc:243] Waited 53 ms on lock 0x7b54003f20d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:37.422088 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:37.422136 (+ 48us) raft_consensus.cc:1685] finished
0503 13:58:37.422245 (+ 109us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:37.422447 (+ 202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":183600,"prepare.queue_time_us":129,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":259359,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":7572,"spinlock_wait_cycles":121853312,"thread_start_us":99088,"threads_started":1,"wal-append.queue_time_us":134995,"wal-append.run_cpu_time_us":1745,"wal-append.run_wall_time_us":1054470}
I20260503 13:58:37.445854 9452 rpcz_store.cc:276] 0503 13:58:36.176099 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:36.403293 (+227194us) service_pool.cc:224] Handling call
0503 13:58:37.414687 (+1011394us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:37.438261 9416 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:37.412894 10721 raft_consensus.cc:740] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:37.369247 10561 consensus_queue.cc:260] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.438388 10794 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.426091 10743 raft_consensus.cc:697] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:37.431977 10706 raft_consensus.cc:3055] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:37.438506 10625 consensus_queue.cc:1048] T 63e583b228f24d97b0e2b31046675c77 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.021s
I20260503 13:58:37.465605 10638 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260503 13:58:37.465930 10558 consensus_queue.cc:1048] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:58:37.466662 9492 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:37.467295 10295 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.468727 10786 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.468914 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 456) took 2341 ms. Trace:
I20260503 13:58:37.470939 10561 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:37.463954 10735 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.473974 9521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 461) took 1499 ms. Trace:
I20260503 13:58:37.474327 10564 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:37.474031 10654 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.475005 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.475997 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.500900 9643 ts_tablet_manager.cc:1434] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.377s user 0.007s sys 0.001s
I20260503 13:58:37.476648 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.476971 10744 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.477413 10463 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.478614 9600 raft_consensus.cc:2410] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260503 13:58:37.481400 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 122001us
I20260503 13:58:37.481712 9598 raft_consensus.cc:1275] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260503 13:58:37.482939 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.492837 10638 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:37.483170 10745 raft_consensus.cc:2749] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260503 13:58:37.486037 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260503 13:58:37.474161 9567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:37.510026 10743 consensus_queue.cc:237] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:37.513280 9567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:37.497797 9448 rpcz_store.cc:276] 0503 13:58:35.127279 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.131454 (+ 4175us) service_pool.cc:224] Handling call
0503 13:58:36.890707 (+1759253us) spinlock_profiling.cc:243] Waited 5.16 ms on lock 0x7b5800000998. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:37.265018 (+374311us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:37.266047 (+ 1029us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:37.266095 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:37.267128 (+ 1033us) log.cc:844] Serialized 26 byte log entry
0503 13:58:37.331230 (+ 64102us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:37.332492 (+ 1262us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:37.335779 (+ 3287us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:37.468480 (+132701us) raft_consensus.cc:1685] finished
0503 13:58:37.468500 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:37.468743 (+ 243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53487,"raft.queue_time_us":45222,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":21803,"spinlock_wait_cycles":11868928,"thread_start_us":59526,"threads_started":1,"wal-append.queue_time_us":181383}
I20260503 13:58:37.513592 10735 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.513907 10744 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.514104 10463 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.515767 10744 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.516031 10463 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.516901 10733 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.517047 9602 raft_consensus.cc:2393] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 6: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:37.517078 9597 raft_consensus.cc:2393] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:37.501749 10295 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.502925 10721 consensus_queue.cc:260] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:37.503703 9522 raft_consensus.cc:1275] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260503 13:58:37.505004 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:37.500521 9521 rpcz_store.cc:276] 0503 13:58:35.974817 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.975522 (+ 705us) service_pool.cc:224] Handling call
0503 13:58:35.989847 (+ 14325us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:35.990152 (+ 305us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:35.990179 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:35.990569 (+ 390us) log.cc:844] Serialized 26 byte log entry
0503 13:58:36.386095 (+395526us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:36.386135 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:36.386201 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:37.396169 (+1009968us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x7b24002aa748. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e8dfe94 00007f9f3e8ddf9b 00007f9f3e96cd39 00007f9f3e972c75 00007f9f3e972b42 00007f9f3e972bce 00007f9f3e3a661b 00007f9f46ae30ba 00007f9f3e972adc 00007f9f3e96519d
0503 13:58:37.473475 (+ 77306us) raft_consensus.cc:1685] finished
0503 13:58:37.473501 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:37.473790 (+ 289us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4491,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":4562,"spinlock_wait_cycles":7039232,"thread_start_us":353162,"threads_started":1,"wal-append.queue_time_us":1177490}
I20260503 13:58:37.508385 9528 raft_consensus.cc:1240] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5. Ops: []
W20260503 13:58:37.508688 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 70580.1us
I20260503 13:58:37.482894 10755 leader_election.cc:290] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.501581 10631 raft_consensus.cc:2749] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260503 13:58:37.487792 10706 raft_consensus.cc:740] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:37.505203 10793 raft_consensus.cc:2804] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:37.543114 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.543658 9643 tablet_bootstrap.cc:492] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
W20260503 13:58:37.543816 10660 consensus_meta.cc:287] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 1.200s user 0.013s sys 0.003s
I20260503 13:58:37.544157 10793 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:37.544589 10793 raft_consensus.cc:3060] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:37.506654 10587 raft_consensus.cc:493] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.515729 10735 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:37.510874 9491 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.26.195:39997 timed out after 1.933s (SENT) [suppressed 44 similar messages]
I20260503 13:58:37.519610 10638 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:37.519484 9598 pending_rounds.cc:85] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868: Aborting all ops after (but not including) 0
I20260503 13:58:37.550607 9598 pending_rounds.cc:107] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260503 13:58:37.549754 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.550840 9598 raft_consensus.cc:2889] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:37.558110 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.558403 9566 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:37.558869 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.560482 9598 raft_consensus.cc:2468] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:37.559321 10721 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:37.509104 10654 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.538230 10706 consensus_queue.cc:260] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.571872 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.548271 10786 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.546440 10733 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.572438 9598 raft_consensus.cc:2468] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
W20260503 13:58:37.582661 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 71804us
I20260503 13:58:37.545152 10564 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:37.593991 9449 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 501) took 1383 ms (client timeout 1573 ms). Trace:
W20260503 13:58:37.551386 9446 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 504) took 1355 ms (client timeout 1806 ms). Trace:
W20260503 13:58:37.595515 9449 rpcz_store.cc:269] 0503 13:58:36.195535 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:37.556587 (+1361052us) service_pool.cc:224] Handling call
0503 13:58:37.578423 (+ 21836us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:37.604506 10706 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:37.585738 10668 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:37.577361 10735 raft_consensus.cc:2749] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:37.616847 9596 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:37.592911 10734 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.575346 10587 raft_consensus.cc:515] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.595041 10631 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:37.639788 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:37.640676 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 45238.9us
I20260503 13:58:37.616721 10445 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.642936 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.643141 10445 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:37.597772 9446 rpcz_store.cc:269] 0503 13:58:36.196171 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:37.476194 (+1280023us) service_pool.cc:224] Handling call
0503 13:58:37.551326 (+ 75132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:37.612917 9595 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:37.615419 10794 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.652137 9447 raft_consensus.cc:1275] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260503 13:58:37.655388 10756 consensus_meta.cc:287] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 1.193s user 0.004s sys 0.013s
I20260503 13:58:37.655723 10649 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.656965 10655 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:37.657492 10655 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.657883 10649 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:37.658986 9600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 429) took 2883 ms (client timeout 1933 ms). Trace:
W20260503 13:58:37.659353 9600 rpcz_store.cc:269] 0503 13:58:34.775107 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.127604 (+352497us) service_pool.cc:224] Handling call
0503 13:58:37.658709 (+2531105us) spinlock_profiling.cc:243] Waited 2.29 s on lock 0x7b54002616d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:37.658889 (+ 180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5259819264}
I20260503 13:58:37.659847 10770 raft_consensus.cc:493] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.660229 10463 raft_consensus.cc:493] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.661005 9599 raft_consensus.cc:1275] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260503 13:58:37.660781 10463 raft_consensus.cc:515] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.661321 9599 pending_rounds.cc:85] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868: Aborting all ops after (but not including) 2
I20260503 13:58:37.659127 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.661700 10631 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.662443 9453 raft_consensus.cc:2468] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:37.663645 10631 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.661674 9599 pending_rounds.cc:107] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260503 13:58:37.656965 9416 leader_election.cc:304] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:37.667079 9599 raft_consensus.cc:2889] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:37.667615 10660 raft_consensus.cc:2749] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:37.664456 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.665505 9594 raft_consensus.cc:2393] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 5: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:37.670321 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:37.653961 10170 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260503 13:58:37.674374 10734 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.676807 9644 tablet_bootstrap.cc:492] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6: No bootstrap required, opened a new log
W20260503 13:58:37.677464 9418 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:37.678601 10660 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.679090 10660 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.679476 10631 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:37.680765 9445 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:37.681070 10660 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.683977 9643 tablet_bootstrap.cc:654] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:37.677412 9644 ts_tablet_manager.cc:1403] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6: Time spent bootstrapping tablet: real 1.728s user 0.025s sys 0.002s
I20260503 13:58:37.661415 10786 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.685981 9416 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:37.680069 10631 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.662410 10655 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.648787 10798 consensus_queue.cc:1048] T f3d3814bacf748a4928419aa5e860f4a P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260503 13:58:37.703346 9600 raft_consensus.cc:2468] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:37.681558 10802 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:37.704295 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.703926 10802 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.705832 9600 raft_consensus.cc:2468] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
W20260503 13:58:37.706197 10812 log.cc:927] Time spent T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260503 13:58:37.660419 10770 raft_consensus.cc:515] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.714277 10719 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.670784 10794 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.723603 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.728878 9600 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:37.733742 10786 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:37.735028 10544 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader election won for term 6
I20260503 13:58:37.735560 10544 raft_consensus.cc:697] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 6 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:37.735304 10786 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.728814 10564 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.737498 10786 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.737174 10544 consensus_queue.cc:237] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.741081 10445 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.744261 9642 tablet_bootstrap.cc:654] T 754ac001a92e4dd6a0b6620803b83722 P 428b342dc4f041df956015e7d7cd173a: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:37.685544 10756 consensus_queue.cc:1048] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
I20260503 13:58:37.751206 9596 raft_consensus.cc:2468] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:37.750993 10793 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.752315 10722 raft_consensus.cc:3055] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:37.755621 9521 raft_consensus.cc:1240] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:37.759541 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.760317 10769 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.746444 10631 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.760489 10445 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.779510 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.780121 10636 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:37.781692 9519 raft_consensus.cc:2393] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:37.770040 10722 raft_consensus.cc:740] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:37.775144 10756 consensus_queue.cc:1243] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer 1cb63886040047559d3893923cc9aaf6 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260503 13:58:37.787608 9524 raft_consensus.cc:2468] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:37.788983 10445 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:37.789685 10804 log.cc:927] Time spent T d5c2e56fa22a496280f2678cd6657c3d P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.062s user 0.003s sys 0.000s
I20260503 13:58:37.797788 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:37.814158 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.800186 10638 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.786870 10654 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:37.793448 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 125532us
I20260503 13:58:37.794502 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.796815 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.799117 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.775272 10769 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:37.804237 10816 log.cc:927] Time spent T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260503 13:58:37.806942 10445 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.808066 10618 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:37.813897 10794 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.813632 10722 consensus_queue.cc:260] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.783735 10743 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.814365 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.822477 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:37.831585 10743 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.832056 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.832716 9597 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 7.
I20260503 13:58:37.787451 10636 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.831131 9598 raft_consensus.cc:2468] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:37.809404 10587 leader_election.cc:290] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:37.834683 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:37.837393 10743 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.837802 9521 raft_consensus.cc:2393] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:37.803016 10770 leader_election.cc:290] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.840694 9527 raft_consensus.cc:2468] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:37.841699 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:37.842233 9522 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:37.843508 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.844112 9525 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 6.
I20260503 13:58:37.845147 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:37.845851 9525 raft_consensus.cc:2393] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:37.845770 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.846364 9593 raft_consensus.cc:1275] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260503 13:58:37.846503 9519 raft_consensus.cc:2393] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:37.846679 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.846956 9593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 437) took 2576 ms. Trace:
I20260503 13:58:37.847193 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.847419 9525 raft_consensus.cc:2393] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 7: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
W20260503 13:58:37.847704 10193 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260503 13:58:37.847927 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.848523 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.849175 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.850526 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.851600 9527 raft_consensus.cc:2410] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:37.845258 9528 raft_consensus.cc:1275] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260503 13:58:37.852828 9528 pending_rounds.cc:85] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a: Aborting all ops after (but not including) 3
I20260503 13:58:37.853102 9528 pending_rounds.cc:107] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260503 13:58:37.853329 9528 raft_consensus.cc:2889] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:37.854724 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.855074 10769 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.856081 10722 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:37.854743 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.858166 10770 raft_consensus.cc:493] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.858667 10673 raft_consensus.cc:493] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.858704 10770 raft_consensus.cc:515] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.859514 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.860157 9596 raft_consensus.cc:2468] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:37.860868 10770 leader_election.cc:290] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.860872 10654 consensus_queue.cc:1048] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260503 13:58:37.863484 10770 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.864529 10770 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:37.847350 9593 rpcz_store.cc:276] 0503 13:58:35.270341 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:35.574810 (+304469us) service_pool.cc:224] Handling call
0503 13:58:35.628178 (+ 53368us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:37.357780 (+1729602us) spinlock_profiling.cc:243] Waited 8.11 ms on lock 0x7b54003efba0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41aa0dfd 00007f9f41a87584 00007f9f41b09827 00007f9f41b223c7 00007f9f41b23bbe 00007f9f41b2df29 00007f9f41b2e7f6 00007f9f41b2b16f 00007f9f41b2ac1e 00007f9f46afdad9
0503 13:58:37.846695 (+488915us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:37.846894 (+ 199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18661760}
I20260503 13:58:37.864730 10673 raft_consensus.cc:515] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.858476 9521 raft_consensus.cc:2468] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:37.816045 10793 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.852010 9520 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:37.880879 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.856092 9601 raft_consensus.cc:2468] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:37.892164 9596 raft_consensus.cc:2468] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:37.882078 9451 consensus_queue.cc:260] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.893821 9449 raft_consensus.cc:1275] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:37.894304 9596 raft_consensus.cc:1275] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:37.891070 9644 raft_consensus.cc:359] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.895325 9644 raft_consensus.cc:385] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:37.895599 9596 raft_consensus.cc:1275] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:37.895998 9519 raft_consensus.cc:2410] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 5 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260503 13:58:37.903083 10561 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.903576 10561 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
W20260503 13:58:37.904872 9565 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:37.905455 10561 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:37.905649 9599 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:37.906435 10193 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.906868 10193 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.908610 10193 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.908895 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.909727 10193 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.909931 10769 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.910249 10193 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.910346 10769 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
@ 0x7f9f3e117a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260503 13:58:37.912207 10769 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.895603 9644 raft_consensus.cc:740] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Initialized, Role: FOLLOWER
I20260503 13:58:37.913628 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.919553 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:37.920130 9597 raft_consensus.cc:2468] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:37.872964 10673 leader_election.cc:290] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.872213 10636 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.924916 9644 consensus_queue.cc:260] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.931146 9522 raft_consensus.cc:2410] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260503 13:58:37.916348 10770 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:37.940919 10770 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:37.941408 10770 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.929107 10793 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.944142 10793 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.944676 10544 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:37.945210 10544 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.912484 10193 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.950003 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.950380 10769 consensus_queue.cc:1048] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:37.925144 10561 raft_consensus.cc:493] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.910848 10295 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.951587 9523 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 6.
I20260503 13:58:37.953578 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.956676 10770 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.956939 9526 raft_consensus.cc:3060] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:37.957222 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.960006 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.944456 10721 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:37.958273 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.964715 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.965149 9521 raft_consensus.cc:2468] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:37.966212 10721 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.967262 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.968241 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.968988 9525 raft_consensus.cc:2393] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:37.969835 9520 raft_consensus.cc:1275] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:37.966604 10638 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:37.976123 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.976727 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.976895 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.977754 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:37.977628 9644 ts_tablet_manager.cc:1434] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6: Time spent starting tablet: real 0.292s user 0.013s sys 0.005s
I20260503 13:58:37.979836 9601 raft_consensus.cc:1275] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260503 13:58:37.983961 9523 raft_consensus.cc:3060] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:37.984771 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.958956 10561 raft_consensus.cc:515] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:37.986850 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d04573f959de406a9590a1603f633e7e" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:37.987648 9444 raft_consensus.cc:2393] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 7: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:37.990250 9567 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:37.987768 10463 leader_election.cc:290] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:37.992594 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:37.993049 9528 raft_consensus.cc:2393] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:37.994026 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.995086 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.980007 10587 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:37.995574 9453 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:37.996732 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.998186 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:37.999899 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.000648 9520 raft_consensus.cc:2410] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260503 13:58:37.995124 10561 leader_election.cc:290] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:38.000989 9525 raft_consensus.cc:3060] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:37.991094 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.004760 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.005549 10654 raft_consensus.cc:2804] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:38.005949 10654 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.006284 10654 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:38.008769 10587 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.011170 10587 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:38.011425 10830 log.cc:927] Time spent T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260503 13:58:38.011749 9643 tablet_bootstrap.cc:492] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:38.012205 9643 ts_tablet_manager.cc:1403] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.469s user 0.020s sys 0.014s
I20260503 13:58:38.012295 10793 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:38.012513 9453 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:38.011685 9444 raft_consensus.cc:2468] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:38.013626 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.014205 9453 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 7.
I20260503 13:58:38.014458 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.017293 10721 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:38.018627 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.021575 10544 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:38.057958 9567 leader_election.cc:304] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:38.064102 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.065042 9453 raft_consensus.cc:2410] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:38.072477 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:38.091753 10561 raft_consensus.cc:2804] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:38.092137 10561 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:38.092480 10561 raft_consensus.cc:3060] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:38.096777 9520 raft_consensus.cc:1240] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:38.098284 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.099344 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.114534 9448 raft_consensus.cc:2468] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
W20260503 13:58:38.116233 10825 log.cc:927] Time spent T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.175s user 0.001s sys 0.001s
W20260503 13:58:38.089779 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 84332.6us
I20260503 13:58:38.158219 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.158195 9523 raft_consensus.cc:2468] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:38.159346 9642 tablet_bootstrap.cc:492] T 754ac001a92e4dd6a0b6620803b83722 P 428b342dc4f041df956015e7d7cd173a: No bootstrap required, opened a new log
I20260503 13:58:38.159811 9642 ts_tablet_manager.cc:1403] T 754ac001a92e4dd6a0b6620803b83722 P 428b342dc4f041df956015e7d7cd173a: Time spent bootstrapping tablet: real 0.733s user 0.022s sys 0.014s
I20260503 13:58:38.174431 9526 raft_consensus.cc:1275] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:38.072630 10544 raft_consensus.cc:493] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:38.197948 9525 raft_consensus.cc:2468] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:38.199617 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:38.200304 9447 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
W20260503 13:58:38.201354 9567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:38.206076 10544 raft_consensus.cc:515] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.208954 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.210019 9643 raft_consensus.cc:359] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.210711 9643 raft_consensus.cc:385] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:38.210968 9643 raft_consensus.cc:740] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:38.204952 10769 raft_consensus.cc:493] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:38.205157 10649 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:38.214399 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.215827 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.108247 10561 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.213008 9643 consensus_queue.cc:260] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.228785 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.233208 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.233925 9453 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:38.240885 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.241552 9525 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 6.
W20260503 13:58:38.242527 10829 log.cc:927] Time spent T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.169s user 0.000s sys 0.001s
W20260503 13:58:38.245054 10826 log.cc:927] Time spent T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.304s user 0.011s sys 0.008s
I20260503 13:58:38.248039 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:38.249024 9447 raft_consensus.cc:2468] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:38.249101 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.249819 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030fcc1b19c3489ea0ca320576ed54ee" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.250396 9520 raft_consensus.cc:2468] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:38.251233 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.251967 9523 raft_consensus.cc:2410] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260503 13:58:38.251834 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.252348 9521 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 6.
I20260503 13:58:38.252737 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.253291 9519 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 6.
I20260503 13:58:38.254132 9567 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:38.255268 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.220001 10649 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.256904 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.249840 9450 raft_consensus.cc:2393] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
W20260503 13:58:38.256194 10834 log.cc:927] Time spent T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.163s user 0.004s sys 0.006s
I20260503 13:58:38.244059 9642 raft_consensus.cc:359] T 754ac001a92e4dd6a0b6620803b83722 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.259006 9642 raft_consensus.cc:385] T 754ac001a92e4dd6a0b6620803b83722 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:38.259302 9642 raft_consensus.cc:740] T 754ac001a92e4dd6a0b6620803b83722 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Initialized, Role: FOLLOWER
I20260503 13:58:38.264916 9642 consensus_queue.cc:260] T 754ac001a92e4dd6a0b6620803b83722 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.268126 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.268669 9520 raft_consensus.cc:2393] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:38.218206 10769 raft_consensus.cc:515] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.270097 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.270705 9527 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:38.271385 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.272747 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.274335 9642 ts_tablet_manager.cc:1434] T 754ac001a92e4dd6a0b6620803b83722 P 428b342dc4f041df956015e7d7cd173a: Time spent starting tablet: real 0.114s user 0.006s sys 0.001s
I20260503 13:58:38.271891 9527 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 6.
I20260503 13:58:38.276780 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.277302 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.278707 9643 ts_tablet_manager.cc:1434] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.264s user 0.011s sys 0.003s
I20260503 13:58:38.255097 10654 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.266896 10649 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:38.280184 9528 raft_consensus.cc:1275] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:38.280982 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.282936 9418 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.281138 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.281365 9525 raft_consensus.cc:1275] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch)
I20260503 13:58:38.280252 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.284814 9453 raft_consensus.cc:2468] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:38.284996 9521 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:38.281654 9523 raft_consensus.cc:1275] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260503 13:58:38.286302 9527 raft_consensus.cc:2468] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:38.284291 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.288066 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.290035 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.292541 10771 raft_consensus.cc:493] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:38.292686 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.293284 9450 raft_consensus.cc:2468] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:38.293051 10771 raft_consensus.cc:515] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.294035 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.295336 9523 raft_consensus.cc:1275] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260503 13:58:38.294535 9450 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 6.
I20260503 13:58:38.297017 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.297927 9418 leader_election.cc:304] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.299085 10459 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260503 13:58:38.299471 10459 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.299772 10459 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Advancing to term 8
I20260503 13:58:38.302309 9452 raft_consensus.cc:1217] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-5.3] Dedup: 3.2->[5.3-5.3]
I20260503 13:58:38.302831 9452 raft_consensus.cc:1240] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20260503 13:58:38.302928 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.297516 9528 raft_consensus.cc:2468] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 0.
I20260503 13:58:38.305241 9521 raft_consensus.cc:1275] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260503 13:58:38.294562 10654 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:38.289885 10802 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:38.305801 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030fcc1b19c3489ea0ca320576ed54ee" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.290813 10618 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260503 13:58:38.307826 9453 raft_consensus.cc:2468] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:38.310092 9643 tablet_bootstrap.cc:492] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:38.310762 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.310210 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.313745 10769 leader_election.cc:290] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:38.314443 10544 leader_election.cc:290] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:38.298241 10771 leader_election.cc:290] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:38.308866 10618 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.319044 9449 raft_consensus.cc:2393] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 6: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:38.320828 9447 raft_consensus.cc:2468] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:38.321545 9567 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:38.323115 10654 consensus_queue.cc:1048] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260503 13:58:38.324712 10650 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:38.325273 10650 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.325608 9567 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:38.315487 10660 raft_consensus.cc:2804] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:38.327530 10650 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:38.324491 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.332218 9567 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:38.322566 10730 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:38.330427 9600 raft_consensus.cc:2468] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:38.335044 9567 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:38.329839 10660 raft_consensus.cc:493] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.322772 10618 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Advancing to term 7
W20260503 13:58:38.336228 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 313462us
I20260503 13:58:38.336830 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.338711 9447 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:38.337179 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.331107 10769 raft_consensus.cc:2804] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:38.333295 10654 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260503 13:58:38.340149 9449 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:38.341516 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.338043 10660 raft_consensus.cc:3060] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:38.343331 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:38.345961 9600 raft_consensus.cc:3060] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:38.335862 10295 raft_consensus.cc:2804] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:38.348984 9567 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:38.338923 10756 raft_consensus.cc:2804] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260503 13:58:38.339371 10561 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:38.352494 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.342242 10769 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.342921 10654 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.352756 10756 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.356797 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:38.358814 9447 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:38.358891 9449 raft_consensus.cc:1275] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:38.336602 10730 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.350519 10193 consensus_queue.cc:1048] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260503 13:58:38.347740 10295 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.360162 10654 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:38.346851 10463 consensus_queue.cc:1048] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.015s
I20260503 13:58:38.358189 10769 raft_consensus.cc:3060] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
W20260503 13:58:37.832789 9405 catalog_manager.cc:6134] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Tablet d049f50b1b5745d3b2e3568ed86c16d8 (table test-workload [id=54209d96daad4531bdc33057c81f7430]) was not created within the allowed timeout. Replacing with a new tablet 4ed72b8f495e46bb93363df9579bed61
I20260503 13:58:38.372752 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.360409 10756 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:38.363621 10295 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:38.347977 10802 consensus_queue.cc:1048] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260503 13:58:38.382740 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.366364 10561 raft_consensus.cc:2804] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:38.381511 9450 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
W20260503 13:58:38.387820 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:38.388872 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.385144 9418 leader_election.cc:304] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.360100 10587 consensus_queue.cc:1059] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.014s
I20260503 13:58:38.392485 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.393147 9450 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:38.394702 9492 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.400426 9447 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 7.
I20260503 13:58:38.402195 9418 leader_election.cc:304] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.403808 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030fcc1b19c3489ea0ca320576ed54ee" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.384840 10561 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.405396 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.405947 9450 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 6.
I20260503 13:58:38.408181 9492 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.410302 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.413782 9492 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.414638 9643 tablet_bootstrap.cc:654] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:38.414812 9492 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.410867 9449 raft_consensus.cc:2468] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:38.407220 10733 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:38.418648 10721 raft_consensus.cc:2804] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:38.418972 10673 raft_consensus.cc:2804] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260503 13:58:38.419330 10673 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.422994 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.385016 10771 consensus_queue.cc:1048] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260503 13:58:38.426024 9418 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.408773 10561 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:38.432492 9447 pending_rounds.cc:85] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6: Aborting all ops after (but not including) 1
I20260503 13:58:38.432802 9447 pending_rounds.cc:107] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260503 13:58:38.433032 9447 raft_consensus.cc:2889] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:38.415495 10793 raft_consensus.cc:2804] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:38.419644 10673 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:38.436775 9492 leader_election.cc:304] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.437526 9492 leader_election.cc:304] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:38.438862 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 89920.1us
I20260503 13:58:38.440711 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.442095 9418 leader_election.cc:304] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.424633 10170 raft_consensus.cc:2804] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:38.443560 9567 leader_election.cc:304] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:38.424010 10721 raft_consensus.cc:697] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:38.446111 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.401387 10650 consensus_queue.cc:1048] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260503 13:58:38.447584 9418 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.453766 9418 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.449692 9601 raft_consensus.cc:2468] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 0.
I20260503 13:58:38.454475 9567 leader_election.cc:304] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:38.438305 10793 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:38.443022 10636 raft_consensus.cc:2804] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:38.448117 10445 raft_consensus.cc:2804] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:38.464672 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.464854 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.467013 9598 raft_consensus.cc:2468] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:38.469244 10543 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:38.452323 10733 raft_consensus.cc:2804] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:38.455233 10802 raft_consensus.cc:2804] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:38.476362 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:38.471562 9405 catalog_manager.cc:6134] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Tablet b633f30c57174316b12d468e847ca831 (table test-workload [id=54209d96daad4531bdc33057c81f7430]) was not created within the allowed timeout. Replacing with a new tablet d95e9ddc94da4ecdb2950bb4d421e508
I20260503 13:58:38.475147 9600 raft_consensus.cc:2468] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:38.466300 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.478240 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:38.462054 10793 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:38.464380 10798 raft_consensus.cc:2804] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:38.465153 10636 raft_consensus.cc:493] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.466012 10649 raft_consensus.cc:2804] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:38.466646 10445 raft_consensus.cc:493] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.480526 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:38.489717 9524 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:38.448267 10463 consensus_queue.cc:1048] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260503 13:58:38.469712 10543 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.467252 10587 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260503 13:58:38.457579 10170 raft_consensus.cc:697] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:38.490806 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.492354 9445 raft_consensus.cc:2410] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:38.473629 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:38.496109 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.496376 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.473930 10733 raft_consensus.cc:493] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.476648 10802 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:38.474916 10842 raft_consensus.cc:2804] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:38.481806 10798 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:38.500084 10798 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:38.500667 10639 raft_consensus.cc:2804] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:38.493304 10722 consensus_queue.cc:1048] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.008s
I20260503 13:58:38.501098 10170 consensus_queue.cc:237] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.503634 9451 raft_consensus.cc:3060] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:38.478175 10730 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:38.488785 10636 raft_consensus.cc:3060] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:38.491133 10649 raft_consensus.cc:493] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.510908 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.491307 10445 raft_consensus.cc:3060] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:38.516466 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.501065 10639 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.516816 9449 raft_consensus.cc:2468] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:38.506203 10842 raft_consensus.cc:493] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.505966 10733 raft_consensus.cc:3060] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:38.505784 10802 raft_consensus.cc:3060] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:38.505251 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.506376 9444 raft_consensus.cc:3060] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:38.526084 9491 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:38.528620 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "754ac001a92e4dd6a0b6620803b83722" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:38.530480 9523 raft_consensus.cc:2468] T 754ac001a92e4dd6a0b6620803b83722 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
I20260503 13:58:38.531908 9418 leader_election.cc:304] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.534366 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:38.534924 9449 raft_consensus.cc:3060] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:38.495080 10771 consensus_queue.cc:1048] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260503 13:58:38.514789 10649 raft_consensus.cc:3060] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:38.540961 10851 raft_consensus.cc:2804] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260503 13:58:38.541383 10851 raft_consensus.cc:493] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.541754 10851 raft_consensus.cc:3060] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [term 0 FOLLOWER]: Advancing to term 1
W20260503 13:58:38.539884 9405 catalog_manager.cc:6134] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Tablet 0f951d26e4ed4c3bb307eb276954663b (table test-workload [id=54209d96daad4531bdc33057c81f7430]) was not created within the allowed timeout. Replacing with a new tablet ccfe3d2570d441418f051b7080cc22a0
I20260503 13:58:38.521962 10639 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:38.546823 9491 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:38.553258 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.553792 9599 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:38.556787 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.559691 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.560245 9597 raft_consensus.cc:2468] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:38.560377 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:38.560931 9598 raft_consensus.cc:3060] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:38.530082 10842 raft_consensus.cc:3060] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:38.535995 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.567849 9491 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:38.577165 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.582134 9594 raft_consensus.cc:1275] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:38.539304 9593 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:38.521572 10721 consensus_queue.cc:237] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.555505 10459 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.590241 9598 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 7.
I20260503 13:58:38.583196 9595 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:38.592527 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d04573f959de406a9590a1603f633e7e" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.584519 9601 raft_consensus.cc:1275] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:38.591223 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.603993 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.604619 9599 raft_consensus.cc:2468] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:38.587116 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.611650 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:38.615103 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 177626us
I20260503 13:58:38.617013 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030fcc1b19c3489ea0ca320576ed54ee" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.618007 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.618721 9598 raft_consensus.cc:2393] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:38.622081 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.622294 9492 leader_election.cc:304] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:38.591173 10459 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 8 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:38.644466 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:38.645969 9594 raft_consensus.cc:3060] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:38.644822 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.647251 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.647790 9598 raft_consensus.cc:2468] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:38.661201 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.667585 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.604014 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.678165 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d04573f959de406a9590a1603f633e7e" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:38.661887 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.681293 9601 raft_consensus.cc:2468] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
W20260503 13:58:38.682166 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 205142us
I20260503 13:58:38.738744 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.743233 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.744081 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.744838 9596 raft_consensus.cc:2393] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 8: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:38.676389 10771 raft_consensus.cc:493] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:38.749425 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.747156 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.750430 9598 raft_consensus.cc:2393] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:38.622720 10561 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:38.757275 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 143958us
W20260503 13:58:38.757750 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 189956us
I20260503 13:58:38.693681 10730 raft_consensus.cc:2749] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:38.821759 9566 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:38.828266 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.830770 9491 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
W20260503 13:58:38.748786 10841 log.cc:927] Time spent T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260503 13:58:38.832587 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.836957 9599 raft_consensus.cc:2393] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 6: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:38.838621 10543 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:38.849632 9598 raft_consensus.cc:1240] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:38.850103 9601 raft_consensus.cc:1275] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260503 13:58:38.850888 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.851385 9599 raft_consensus.cc:2393] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 6: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
W20260503 13:58:38.847886 10815 log.cc:927] Time spent T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260503 13:58:38.857219 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:38.856125 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.858649 9526 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Advancing to term 8
I20260503 13:58:38.856863 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.860339 9566 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:38.757499 10771 raft_consensus.cc:515] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.844233 10654 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.864864 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:38.852098 10668 raft_consensus.cc:3055] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:38.853579 10543 raft_consensus.cc:2804] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:38.864233 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:38.872088 10543 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:38.875957 9453 raft_consensus.cc:2410] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260503 13:58:38.878321 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:38.872895 10846 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:38.906141 10846 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:38.909129 10855 raft_consensus.cc:2804] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:38.909560 10855 raft_consensus.cc:493] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:38.909888 10855 raft_consensus.cc:3060] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:38.855502 10861 raft_consensus.cc:2749] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:38.863783 10722 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:38.847462 10732 consensus_queue.cc:1048] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260503 13:58:38.943027 9447 raft_consensus.cc:1275] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260503 13:58:38.871253 10845 log.cc:927] Time spent T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260503 13:58:38.872538 10768 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260503 13:58:38.961818 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:38.872515 10543 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:38.869261 10668 raft_consensus.cc:740] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:39.013844 9599 raft_consensus.cc:1240] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260503 13:58:39.017138 10844 log.cc:927] Time spent T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.345s user 0.000s sys 0.001s
I20260503 13:58:39.018687 9452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 520) took 1010 ms. Trace:
I20260503 13:58:38.944496 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "754ac001a92e4dd6a0b6620803b83722" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:39.019150 9452 rpcz_store.cc:276] 0503 13:58:38.008516 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:38.304680 (+296164us) service_pool.cc:224] Handling call
0503 13:58:38.310069 (+ 5389us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:38.317809 (+ 7740us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:38.331314 (+ 13505us) log.cc:844] Serialized 14 byte log entry
0503 13:58:38.439577 (+108263us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:38.465277 (+ 25700us) log.cc:844] Serialized 26 byte log entry
0503 13:58:38.465398 (+ 121us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:38.465426 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:38.465496 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:39.018484 (+552988us) raft_consensus.cc:1685] finished
0503 13:58:39.018504 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:39.018631 (+ 127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":124,"prepare.queue_time_us":11214,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":69,"raft.queue_time_us":7376,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":9218,"thread_start_us":93185,"threads_started":1,"wal-append.queue_time_us":128440}
I20260503 13:58:39.018577 10847 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:38.969177 10722 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.051744 9599 raft_consensus.cc:1240] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 5. Ops: []
W20260503 13:58:39.063675 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 197771us
I20260503 13:58:39.063870 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 497) took 1002 ms. Trace:
I20260503 13:58:39.064937 9566 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.067948 9452 raft_consensus.cc:1275] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 1. (term mismatch)
I20260503 13:58:39.068434 9452 pending_rounds.cc:85] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6: Aborting all ops after (but not including) 0
I20260503 13:58:39.068688 9452 pending_rounds.cc:107] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260503 13:58:39.068897 9452 raft_consensus.cc:2889] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260503 13:58:39.070026 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 223019us
I20260503 13:58:39.071070 10852 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
W20260503 13:58:39.073374 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:39.069883 9566 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.065160 9448 rpcz_store.cc:276] 0503 13:58:38.062383 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:38.328940 (+266557us) service_pool.cc:224] Handling call
0503 13:58:38.333923 (+ 4983us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:38.340885 (+ 6962us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:38.340910 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:38.347258 (+ 6348us) log.cc:844] Serialized 26 byte log entry
0503 13:58:38.439407 (+ 92149us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:38.439441 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:38.439499 (+ 58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:39.063676 (+624177us) raft_consensus.cc:1685] finished
0503 13:58:39.063695 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:39.063817 (+ 122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1400,"raft.queue_time_us":5458,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":75,"thread_start_us":91827,"threads_started":1,"wal-append.queue_time_us":133093}
I20260503 13:58:39.071686 10852 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.081091 10850 raft_consensus.cc:2749] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
@ 0x7f9f480a6e4e kudu::TestWorkload::Setup() at ??:0
I20260503 13:58:39.115056 10850 raft_consensus.cc:2764] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
@ 0x354cc8 kudu::master::LeaderRebalancerTest::CreateWorkloadTable() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_leader_rebalancer-test.cc:110
I20260503 13:58:38.942982 10856 consensus_queue.cc:1048] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.205s
@ 0x350d2f kudu::master::LeaderRebalancerTest_AddTserver_Test::TestBody() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_leader_rebalancer-test.cc:415
I20260503 13:58:39.113180 10852 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:39.073699 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 451425us
W20260503 13:58:39.107898 10660 consensus_meta.cc:287] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.753s user 0.003s sys 0.003s
I20260503 13:58:39.109055 9525 raft_consensus.cc:1240] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:39.109426 10846 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:39.110329 10618 consensus_meta.cc:287] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.768s user 0.003s sys 0.004s
I20260503 13:58:39.083091 10463 raft_consensus.cc:2804] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:39.075862 10668 consensus_queue.cc:260] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:39.111135 10673 consensus_meta.cc:287] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.658s user 0.005s sys 0.010s
I20260503 13:58:39.075614 10849 raft_consensus.cc:2804] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260503 13:58:39.112279 10793 consensus_meta.cc:287] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.615s user 0.005s sys 0.000s
W20260503 13:58:39.113775 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 43972.4us
W20260503 13:58:39.128068 10857 log.cc:927] Time spent T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.280s user 0.001s sys 0.000s
W20260503 13:58:39.123416 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 292810us
I20260503 13:58:39.050343 9595 raft_consensus.cc:2468] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 0.
W20260503 13:58:39.129269 9524 consensus_meta.cc:287] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.633s user 0.012s sys 0.015s
I20260503 13:58:39.137495 9492 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:39.123273 10170 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.015s sys 0.015s
W20260503 13:58:39.140097 9567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:39.142472 9566 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.173967 9566 leader_election.cc:304] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.174636 9526 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 8.
W20260503 13:58:39.121520 9451 consensus_meta.cc:287] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.614s user 0.019s sys 0.002s
I20260503 13:58:39.125756 10561 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:39.124781 10463 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:39.146946 10793 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.126197 10618 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.140620 10849 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:39.149441 9524 raft_consensus.cc:2468] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:39.174966 9566 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.181092 10852 consensus_queue.cc:1048] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260503 13:58:39.184903 10793 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.182499 10587 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:39.185730 10587 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:39.186630 10872 consensus_queue.cc:1059] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260503 13:58:39.181111 10846 raft_consensus.cc:2749] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:39.188140 10771 leader_election.cc:290] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.179103 10869 raft_consensus.cc:2804] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:39.188975 10862 raft_consensus.cc:2764] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260503 13:58:39.189688 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:39.190331 9566 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.125142 10660 raft_consensus.cc:515] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:39.173098 10170 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.669s user 0.016s sys 0.015s
I20260503 13:58:39.049913 10847 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.135450 10673 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.179324 9492 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:39.179415 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:39.178539 10861 raft_consensus.cc:2749] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:39.180478 9451 raft_consensus.cc:1275] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260503 13:58:39.208667 9451 pending_rounds.cc:85] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6: Aborting all ops after (but not including) 2
I20260503 13:58:39.208913 9451 pending_rounds.cc:107] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260503 13:58:39.209136 9451 raft_consensus.cc:2889] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:39.178869 10463 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
W20260503 13:58:39.180946 10865 log.cc:927] Time spent T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20260503 13:58:39.125973 10654 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:39.199132 10733 consensus_meta.cc:287] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.661s user 0.017s sys 0.015s
W20260503 13:58:39.177276 10769 consensus_meta.cc:287] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.788s user 0.019s sys 0.004s
I20260503 13:58:39.191336 9566 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.197708 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:39.197809 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:39.199365 9643 tablet_bootstrap.cc:492] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:39.206909 9450 raft_consensus.cc:1240] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260503 13:58:39.230578 9643 ts_tablet_manager.cc:1403] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.926s user 0.014s sys 0.021s
I20260503 13:58:39.209681 9526 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 7
I20260503 13:58:39.182961 9567 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:39.193125 10869 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:39.191632 10849 raft_consensus.cc:3060] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:39.229061 9445 raft_consensus.cc:2393] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
W20260503 13:58:39.235116 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 55856.1us
I20260503 13:58:39.237277 9492 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:39.235422 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 444) took 4687 ms. Trace:
W20260503 13:58:39.232968 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 50088us
I20260503 13:58:39.238523 9567 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:39.237763 9451 rpcz_store.cc:276] 0503 13:58:34.541794 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:34.795105 (+253311us) service_pool.cc:224] Handling call
0503 13:58:34.825415 (+ 30310us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:36.382635 (+1557220us) spinlock_profiling.cc:243] Waited 62.3 ms on lock 0x7b54003ac3a0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41aa944f 00007f9f41b22340 00007f9f41b23bbe 00007f9f41b2df29 00007f9f41b2e7f6 00007f9f41b2b16f 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2
0503 13:58:37.881513 (+1498878us) spinlock_profiling.cc:243] Waited 882 ms on lock 0x7b54003ac3a0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41aa8900 00007f9f41b22350 00007f9f41b23bbe 00007f9f41b2df29 00007f9f41b2e7f6 00007f9f41b2b16f 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2
0503 13:58:38.493489 (+611976us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b5800187ad0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41a8748a 00007f9f41b09827 00007f9f41b223c7 00007f9f41b23bbe 00007f9f41b2df29 00007f9f41b2e7f6 00007f9f41b2b16f 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34
0503 13:58:39.224080 (+730591us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:39.229362 (+ 5282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3008187904}
W20260503 13:58:39.232043 9491 consensus_peers.cc:597] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 428b342dc4f041df956015e7d7cd173a -> Peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Couldn't send request to peer 4bea375a55034750a1f8804b0db97868. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260503 13:58:39.241590 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 70741.9us
I20260503 13:58:39.207708 10561 raft_consensus.cc:2749] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260503 13:58:39.211310 10879 raft_consensus.cc:2749] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260503 13:58:39.253273 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 62006.4us
I20260503 13:58:39.257069 9566 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.225808 10855 raft_consensus.cc:515] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.258919 10660 leader_election.cc:290] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.259546 10855 leader_election.cc:290] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.256713 9492 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:39.263094 9599 raft_consensus.cc:1275] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260503 13:58:39.257105 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 135179us
I20260503 13:58:39.231309 10733 raft_consensus.cc:515] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.294458 10618 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.241492 10869 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:39.279893 9492 leader_election.cc:304] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:39.246053 10793 raft_consensus.cc:2804] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260503 13:58:39.249671 10654 raft_consensus.cc:2804] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 8 FOLLOWER]: Leader election won for term 8
W20260503 13:58:39.296139 10636 consensus_meta.cc:287] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.773s user 0.019s sys 0.019s
I20260503 13:58:39.281387 9491 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:39.281437 9566 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:39.283826 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:39.261281 10756 consensus_meta.cc:287] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.872s user 0.010s sys 0.003s
I20260503 13:58:39.252709 10635 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:39.254274 10722 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.264881 9521 raft_consensus.cc:1240] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:39.236734 10861 raft_consensus.cc:2749] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:39.260026 10561 raft_consensus.cc:2804] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:39.236056 10769 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.264673 10862 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260503 13:58:39.291416 9526 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 7.
I20260503 13:58:39.264881 9595 raft_consensus.cc:1240] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:39.293888 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:39.265018 9599 raft_consensus.cc:1240] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 6. Current term is 7. Ops: []
W20260503 13:58:39.293903 10864 log.cc:927] Time spent T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20260503 13:58:39.284698 10650 raft_consensus.cc:2804] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader election won for term 7
W20260503 13:58:39.280489 9416 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:39.296063 10879 raft_consensus.cc:2804] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:39.293843 10668 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:39.323479 9566 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.330391 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:39.307709 10793 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:39.333143 9599 raft_consensus.cc:1240] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260503 13:58:39.264422 10868 consensus_queue.cc:1048] T 3500374093e84b968ed96ed4dbdfda69 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.052s
W20260503 13:58:39.334323 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 53015.1us
I20260503 13:58:39.334723 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:39.332330 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 70223.8us
I20260503 13:58:39.338397 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:39.330845 10802 consensus_meta.cc:287] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.780s user 0.033s sys 0.021s
I20260503 13:58:39.321197 10654 raft_consensus.cc:697] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [term 8 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:39.329739 10770 raft_consensus.cc:2804] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:39.325842 10862 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:39.335665 10793 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:39.369536 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:39.317411 10636 raft_consensus.cc:515] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.370180 9451 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:39.340204 10861 raft_consensus.cc:2764] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260503 13:58:39.375241 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:39.368508 10445 consensus_meta.cc:287] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.855s user 0.005s sys 0.028s
I20260503 13:58:39.378086 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:39.382264 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:39.381336 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:39.384018 9453 raft_consensus.cc:3060] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:39.374447 9567 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:39.321735 10756 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.350844 10879 raft_consensus.cc:697] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
W20260503 13:58:39.386333 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:39.325120 10635 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.352277 10770 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:39.370448 10862 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Advancing to term 7
W20260503 13:58:39.399528 9594 consensus_meta.cc:287] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.722s user 0.007s sys 0.011s
W20260503 13:58:39.399820 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 49110.2us
I20260503 13:58:39.403574 9594 raft_consensus.cc:2468] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:39.399952 9526 raft_consensus.cc:2393] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:39.405033 9416 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:39.405692 10876 log.cc:927] Time spent T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260503 13:58:39.407886 10654 consensus_queue.cc:237] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:39.409042 10873 log.cc:927] Time spent T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.130s user 0.000s sys 0.001s
W20260503 13:58:39.411443 10878 log.cc:927] Time spent T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20260503 13:58:39.416730 10889 raft_consensus.cc:2749] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:39.417369 10463 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.419107 10847 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.419272 10463 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:39.333968 10880 raft_consensus.cc:2804] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:39.342814 10856 consensus_queue.cc:1048] T cefaab375804454f943c82f15b8fd68b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260503 13:58:39.376818 10771 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:39.379760 10673 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.350736 10802 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:39.462105 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 121272us
I20260503 13:58:39.427760 9416 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:39.399528 9449 consensus_meta.cc:287] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.862s user 0.020s sys 0.020s
I20260503 13:58:39.463243 9449 raft_consensus.cc:2468] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:39.429174 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:39.429705 9600 raft_consensus.cc:2393] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 7: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:39.422760 10861 consensus_queue.cc:1059] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260503 13:58:39.464490 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 141065us
I20260503 13:58:39.465272 9566 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.466934 9491 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:39.467031 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:39.467706 9521 raft_consensus.cc:2393] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:39.403373 10770 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:39.468154 9491 leader_election.cc:304] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:39.429244 9523 raft_consensus.cc:1240] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 6. Current term is 7. Ops: []
I20260503 13:58:39.472029 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:39.472579 9528 raft_consensus.cc:3060] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:39.423692 10879 consensus_queue.cc:237] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:39.426965 10649 consensus_meta.cc:287] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.872s user 0.015s sys 0.006s
I20260503 13:58:39.439604 10543 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.440451 10733 leader_election.cc:290] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:39.441440 10854 log.cc:927] Time spent T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.606s user 0.002s sys 0.000s
W20260503 13:58:39.471419 10842 consensus_meta.cc:287] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.890s user 0.007s sys 0.010s
I20260503 13:58:39.443879 10587 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.445444 10636 leader_election.cc:290] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.447813 9450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 530) took 1140 ms. Trace:
I20260503 13:58:39.405213 10884 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:39.405005 10874 log.cc:927] Time spent T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20260503 13:58:39.386623 10445 raft_consensus.cc:515] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.453748 10880 raft_consensus.cc:697] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:39.515555 10445 leader_election.cc:290] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.515339 10880 consensus_queue.cc:237] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:39.516434 10295 consensus_meta.cc:287] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 1.122s user 0.009s sys 0.009s
I20260503 13:58:39.517949 10543 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:39.458159 9444 consensus_meta.cc:287] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.903s user 0.019s sys 0.015s
I20260503 13:58:39.435156 10850 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader election won for term 7
I20260503 13:58:39.442775 9596 raft_consensus.cc:1240] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:39.467010 10561 raft_consensus.cc:697] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:39.502317 9450 rpcz_store.cc:276] 0503 13:58:38.307056 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:38.411733 (+104677us) service_pool.cc:224] Handling call
0503 13:58:38.423989 (+ 12256us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:39.230489 (+806500us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:39.447586 (+217097us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b54001d3654. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:39.447757 (+ 171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1799986048}
W20260503 13:58:39.507172 10639 consensus_meta.cc:287] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.933s user 0.013s sys 0.009s
I20260503 13:58:39.508121 10891 raft_consensus.cc:2764] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260503 13:58:39.510928 10871 raft_consensus.cc:2764] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260503 13:58:39.521515 9453 raft_consensus.cc:2468] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:39.523218 9566 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.487672 10170 raft_consensus.cc:2749] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:39.467351 10771 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.560629 9643 raft_consensus.cc:359] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.561278 9643 raft_consensus.cc:385] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:39.561524 9643 raft_consensus.cc:740] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:39.562131 9643 consensus_queue.cc:260] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.516860 10295 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:39.605397 10673 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:39.605902 10673 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.607650 10673 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:39.610991 10798 consensus_meta.cc:287] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 1.110s user 0.009s sys 0.009s
I20260503 13:58:39.611410 10798 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.612313 9451 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:39.618611 9451 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 7.
I20260503 13:58:39.621716 10798 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.623112 10862 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.625082 10862 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.522101 9444 raft_consensus.cc:2468] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:39.488986 10649 raft_consensus.cc:515] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.556375 10847 raft_consensus.cc:2804] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:39.552030 10850 raft_consensus.cc:697] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 7 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:39.560344 10769 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:39.639829 9602 raft_consensus.cc:2393] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 5: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:39.519745 10842 raft_consensus.cc:515] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.553944 10639 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.632078 10756 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:39.645283 9492 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:39.640045 10847 raft_consensus.cc:493] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:39.646061 10650 raft_consensus.cc:697] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [term 7 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:39.651711 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 436) took 1089 ms. Trace:
I20260503 13:58:39.642206 10768 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
W20260503 13:58:39.654160 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 226458us
I20260503 13:58:39.655298 9522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 514) took 1050 ms. Trace:
I20260503 13:58:39.657188 9522 rpcz_store.cc:276] 0503 13:58:38.604989 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:39.073153 (+468164us) service_pool.cc:224] Handling call
0503 13:58:39.073467 (+ 314us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:39.247007 (+173540us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:39.247039 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:39.260733 (+ 13694us) log.cc:844] Serialized 26 byte log entry
0503 13:58:39.326027 (+ 65294us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:39.326060 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:39.654924 (+328864us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b5400155ed4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:39.654969 (+ 45us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:39.655013 (+ 44us) raft_consensus.cc:1685] finished
0503 13:58:39.655047 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:39.655243 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23768,"raft.queue_time_us":17185,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":7502,"spinlock_wait_cycles":355843840,"thread_start_us":31054,"threads_started":1,"wal-append.queue_time_us":179151}
I20260503 13:58:39.654932 9598 rpcz_store.cc:276] 0503 13:58:38.562558 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:38.832854 (+270296us) service_pool.cc:224] Handling call
0503 13:58:38.833115 (+ 261us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:38.853417 (+ 20302us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:39.651492 (+798075us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b54003f07d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:39.651646 (+ 154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30041856}
I20260503 13:58:39.659149 9416 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.646081 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 515) took 1276 ms. Trace:
I20260503 13:58:39.665686 9525 raft_consensus.cc:1240] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
W20260503 13:58:39.665911 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 197805us
W20260503 13:58:39.642441 9444 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 506) took 1534 ms (client timeout 1701 ms). Trace:
I20260503 13:58:39.667588 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:39.528420 10884 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.664354 9447 rpcz_store.cc:276] 0503 13:58:38.369262 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:38.982250 (+612988us) service_pool.cc:224] Handling call
0503 13:58:38.999809 (+ 17559us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:39.035098 (+ 35289us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:39.067188 (+ 32090us) log.cc:844] Serialized 14 byte log entry
0503 13:58:39.453401 (+386213us) spinlock_profiling.cc:243] Waited 5.49 ms on lock 0x7b180012f580. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea70587 00007f9f47869bc9 00007f9f3ea8ccdb 00007f9f3ea8af7f 00007f9f3ea8aa95 00007f9f4552e837 00007f9f41b35d87 00007f9f41b4549f 00007f9f41b45418
0503 13:58:39.645604 (+192203us) log.cc:844] Serialized 14 byte log entry
0503 13:58:39.645791 (+ 187us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:39.645834 (+ 43us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:39.645868 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:39.645937 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:39.646034 (+ 97us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2194,"prepare.queue_time_us":43422,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":76,"raft.queue_time_us":2378,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":14847,"spinlock_wait_cycles":12622080,"thread_start_us":295980,"threads_started":2,"wal-append.queue_time_us":156938}
W20260503 13:58:39.663985 9416 leader_election.cc:343] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:39.672479 9492 leader_election.cc:304] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:39.674352 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:39.652541 10846 raft_consensus.cc:2749] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260503 13:58:39.673765 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:39.654769 10847 raft_consensus.cc:3060] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:39.654587 10636 raft_consensus.cc:2804] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260503 13:58:39.667153 10851 consensus_meta.cc:287] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 1.116s user 0.015s sys 0.004s
I20260503 13:58:39.664984 10768 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Advancing to term 7
W20260503 13:58:39.690549 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 232723us
W20260503 13:58:39.698601 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:39.714674 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 50706.2us
I20260503 13:58:39.721959 9491 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:39.717649 9448 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:39.725342 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:39.715958 9593 raft_consensus.cc:2393] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:39.720890 10636 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:39.731705 10636 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:39.685983 10862 raft_consensus.cc:2804] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election won for term 2
I20260503 13:58:39.671240 10445 consensus_queue.cc:1048] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
W20260503 13:58:39.736361 9416 leader_election.cc:343] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:39.738083 9416 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:39.686846 10802 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.740386 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:39.741052 9447 raft_consensus.cc:2468] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:39.741287 10869 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:39.741917 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:39.742236 10650 consensus_queue.cc:237] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.742789 10654 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:39.743348 10654 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.734761 10871 consensus_queue.cc:1048] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260503 13:58:39.745317 10654 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:39.745679 10875 raft_consensus.cc:2749] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260503 13:58:39.723104 10850 consensus_queue.cc:237] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.749439 10918 consensus_queue.cc:1048] T 0db0ab3576734ec18108a10f65e31341 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:39.750206 10770 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.750571 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:39.752105 10770 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.721349 10889 raft_consensus.cc:493] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:39.752723 10889 raft_consensus.cc:515] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.755240 9447 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:39.755966 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:39.781546 9527 raft_consensus.cc:2393] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
W20260503 13:58:39.672726 9444 rpcz_store.cc:269] 0503 13:58:38.104983 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:38.468948 (+363965us) service_pool.cc:224] Handling call
0503 13:58:39.634208 (+1165260us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:39.789654 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 104605us
W20260503 13:58:39.802255 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 279101us
W20260503 13:58:39.852684 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 113134us
I20260503 13:58:39.806813 9492 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:39.672201 10561 consensus_queue.cc:237] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:39.861227 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 123217us
I20260503 13:58:39.855746 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:39.895444 9524 raft_consensus.cc:2393] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:39.895349 9416 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:39.855233 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 48506us
I20260503 13:58:39.896884 9566 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.896220 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:39.897856 9450 raft_consensus.cc:2393] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:39.902855 9643 ts_tablet_manager.cc:1434] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.672s user 0.006s sys 0.014s
I20260503 13:58:39.720366 10851 raft_consensus.cc:515] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.911298 10896 raft_consensus.cc:2749] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:39.912637 10851 leader_election.cc:290] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.913196 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030fcc1b19c3489ea0ca320576ed54ee" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:39.913946 9491 leader_election.cc:304] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:39.915704 9491 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:39.916441 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:39.916993 9450 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
W20260503 13:58:39.917359 10917 log.cc:927] Time spent T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20260503 13:58:39.789422 10862 raft_consensus.cc:697] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:39.781224 10587 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.860550 10770 raft_consensus.cc:2804] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260503 13:58:39.859462 10849 consensus_meta.cc:287] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.605s user 0.007s sys 0.000s
I20260503 13:58:39.936304 9447 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 7.
I20260503 13:58:39.913942 9444 raft_consensus.cc:2393] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
W20260503 13:58:39.904713 9643 ts_tablet_manager.cc:1467] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868: Tablet startup took 673ms
I20260503 13:58:39.938130 9566 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.857877 10897 raft_consensus.cc:2749] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260503 13:58:39.939666 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 44407.7us
I20260503 13:58:39.854116 10721 raft_consensus.cc:2749] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260503 13:58:39.958503 9597 raft_consensus.cc:1240] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 5. Ops: []
I20260503 13:58:39.790719 10756 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:39.945075 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:39.939103 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:39.956475 9416 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:39.918901 10649 leader_election.cc:290] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.931452 10771 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.858445 10793 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.938062 10295 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
I20260503 13:58:39.947233 10770 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:39.945612 10638 raft_consensus.cc:2804] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:39.856482 10445 consensus_queue.cc:1059] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.107s
I20260503 13:58:39.947042 10911 raft_consensus.cc:2749] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:39.944217 10912 raft_consensus.cc:2804] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:39.962110 10639 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:39.962826 10851 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
W20260503 13:58:40.033016 10921 log.cc:927] Time spent T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
W20260503 13:58:38.830308 9405 catalog_manager.cc:6134] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Tablet f3d3814bacf748a4928419aa5e860f4a (table test-workload [id=54209d96daad4531bdc33057c81f7430]) was not created within the allowed timeout. Replacing with a new tablet 699cd55356e945478a6197e5bcbecfa7
W20260503 13:58:40.053786 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:40.054013 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260503 13:58:40.055217 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 157745us
I20260503 13:58:40.058027 9520 raft_consensus.cc:2393] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 7: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
W20260503 13:58:40.058645 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 113574us
I20260503 13:58:40.059404 9491 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:39.955237 10722 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
W20260503 13:58:39.968147 9602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 481) took 1540 ms (client timeout 1848 ms). Trace:
I20260503 13:58:40.067407 9492 leader_election.cc:304] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:40.029227 10770 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
W20260503 13:58:40.066959 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 110556us
I20260503 13:58:40.069648 9416 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:40.066706 9602 rpcz_store.cc:269] 0503 13:58:38.418065 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:38.418191 (+ 126us) service_pool.cc:224] Handling call
0503 13:58:39.958111 (+1539920us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400260cd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:39.958242 (+ 131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2796674944}
W20260503 13:58:40.071625 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 133568us
I20260503 13:58:40.029873 10638 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:40.076329 10722 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.078321 10722 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.079125 10445 raft_consensus.cc:3055] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 6 LEADER]: Stepping down as leader of term 6
I20260503 13:58:40.081038 9566 leader_election.cc:304] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:39.960171 10842 leader_election.cc:290] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:40.079648 10445 raft_consensus.cc:740] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:39.948482 10543 consensus_queue.cc:1048] T 8cc630de8e5547ceb5c178981ee99fc2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.118s
I20260503 13:58:40.061710 10544 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:40.024775 10756 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.067426 10889 leader_election.cc:290] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:40.053552 10912 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:40.074594 10721 raft_consensus.cc:2749] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:40.074774 10295 raft_consensus.cc:2749] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260503 13:58:40.082106 10638 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:40.061553 10170 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:40.111816 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 539) took 1629 ms. Trace:
I20260503 13:58:40.073153 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:40.114192 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 44634.5us
I20260503 13:58:40.113523 10544 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
W20260503 13:58:40.127063 9416 leader_election.cc:343] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:40.127884 9416 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:40.128298 10544 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.128551 10886 raft_consensus.cc:2749] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:40.118228 10912 raft_consensus.cc:3060] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:40.129755 10793 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.124392 10802 raft_consensus.cc:2804] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:40.093575 10730 consensus_queue.cc:1048] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.037s
I20260503 13:58:40.130378 10802 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
@ 0x7f9f3ec7fe30 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
I20260503 13:58:40.130721 10802 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:40.126128 9445 rpcz_store.cc:276] 0503 13:58:38.482164 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:38.495301 (+ 13137us) service_pool.cc:224] Handling call
0503 13:58:40.111646 (+1616345us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x7b54001a04d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:40.111766 (+ 120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1643259776}
W20260503 13:58:39.953929 9643 ts_tablet_manager.cc:1469] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868: Trace:
0503 13:58:05.693223 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.291825 (+598602us) service_pool.cc:224] Handling call
0503 13:58:06.307253 (+ 15428us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.307349 (+ 96us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.391587 (+ 84238us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.427871 (+ 36284us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.428265 (+ 394us) inbound_call.cc:177] Queueing success response
0503 13:58:38.291550 (+31863285us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0503 13:58:39.230887 (+939337us) ts_tablet_manager.cc:1435] Starting tablet replica
0503 13:58:39.428294 (+197407us) tablet_replica.cc:261] Starting instrumentation
0503 13:58:39.436043 (+ 7749us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":59870,"raft.queue_time_us":88714,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":16498,"tablet-open.queue_time_us":31854373,"thread_start_us":169882,"threads_started":1}
W20260503 13:58:40.116901 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 44618.5us
I20260503 13:58:40.128883 9416 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:40.088457 9450 raft_consensus.cc:2468] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:40.133769 9566 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:40.119287 9528 consensus_meta.cc:287] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.629s user 0.007s sys 0.001s
I20260503 13:58:40.062135 10851 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:39.955536 10849 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.134866 9528 raft_consensus.cc:2468] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:40.135702 10925 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:40.136297 10869 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997), 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599)
W20260503 13:58:40.136636 10904 log.cc:927] Time spent T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.191s user 0.000s sys 0.000s
I20260503 13:58:40.087358 10927 raft_consensus.cc:2749] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:40.165315 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:40.143755 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 76396.1us
I20260503 13:58:40.144764 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:40.144889 9416 leader_election.cc:304] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:40.148809 10721 raft_consensus.cc:2749] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:40.148998 10925 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.149374 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030fcc1b19c3489ea0ca320576ed54ee" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.173353 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030fcc1b19c3489ea0ca320576ed54ee" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:40.174089 9526 raft_consensus.cc:2393] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:40.175056 9567 leader_election.cc:304] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:40.175774 9519 raft_consensus.cc:1240] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:40.150837 10851 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.157052 9566 leader_election.cc:304] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:40.161314 10907 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:40.141002 10879 raft_consensus.cc:2749] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260503 13:58:40.138216 10884 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.106696 10847 raft_consensus.cc:515] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.168226 9416 leader_election.cc:304] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:40.171036 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 111708us
I20260503 13:58:40.180975 9491 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:40.171667 9528 raft_consensus.cc:2393] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:40.178779 9521 raft_consensus.cc:1275] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260503 13:58:40.182515 9416 leader_election.cc:304] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:40.123962 10842 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:40.142073 9444 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:40.186394 9643 tablet_bootstrap.cc:492] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:40.183709 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:40.139851 10170 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.189620 10170 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.126343 10445 consensus_queue.cc:260] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.192590 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.193266 9521 raft_consensus.cc:2393] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:40.193501 10891 raft_consensus.cc:3055] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:40.193871 10891 raft_consensus.cc:740] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:40.194070 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "754ac001a92e4dd6a0b6620803b83722" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:40.194579 9521 raft_consensus.cc:3060] T 754ac001a92e4dd6a0b6620803b83722 P 428b342dc4f041df956015e7d7cd173a [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:40.194823 10925 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:40.173372 10916 raft_consensus.cc:2804] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:40.188367 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.166513 10639 raft_consensus.cc:2804] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader election won for term 7
I20260503 13:58:40.208328 10916 raft_consensus.cc:493] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:40.208653 9523 raft_consensus.cc:2393] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:40.195408 10920 raft_consensus.cc:2804] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:40.193197 10445 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:40.209574 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.210098 9523 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:40.210712 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:40.213061 10869 raft_consensus.cc:2804] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:40.213406 10869 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:40.213606 9522 raft_consensus.cc:1240] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:40.213760 10869 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:40.195955 10884 raft_consensus.cc:2749] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:40.214532 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 569) took 1520 ms. Trace:
I20260503 13:58:40.214645 10897 consensus_queue.cc:1048] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260503 13:58:40.214877 9525 rpcz_store.cc:276] 0503 13:58:38.694120 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:39.125064 (+430944us) service_pool.cc:224] Handling call
0503 13:58:39.137655 (+ 12591us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:39.666033 (+528378us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:40.214313 (+548280us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x7b540004d654. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:40.214481 (+ 168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1213842816}
I20260503 13:58:40.215926 10635 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:40.216140 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:40.208575 10639 raft_consensus.cc:697] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [term 7 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:40.199721 10636 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.201347 10849 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.201236 10891 consensus_queue.cc:260] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.202282 10721 raft_consensus.cc:3060] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Advancing to term 8
W20260503 13:58:40.203586 10853 log.cc:927] Time spent T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 1.367s user 0.005s sys 0.015s
I20260503 13:58:40.206332 10912 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.179992 10649 raft_consensus.cc:2749] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260503 13:58:40.207044 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.218380 9416 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:40.241986 9416 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:40.208798 10916 raft_consensus.cc:3060] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:40.227730 9566 leader_election.cc:304] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:40.214239 9491 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:40.210155 10847 leader_election.cc:290] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.227641 9492 leader_election.cc:304] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:40.088629 10862 consensus_queue.cc:237] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.233857 9418 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:40.216502 10896 raft_consensus.cc:2749] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:40.245224 9491 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:40.248342 9416 leader_election.cc:304] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:40.229794 10193 raft_consensus.cc:493] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:40.250347 9492 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:40.217195 10850 consensus_queue.cc:1048] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.022s
I20260503 13:58:40.256325 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:40.257082 9491 leader_election.cc:343] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:40.269451 9491 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:40.257617 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.228682 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:40.219219 10851 consensus_queue.cc:1048] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260503 13:58:40.270427 9528 raft_consensus.cc:2368] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for earlier term 6. Current term is 7.
I20260503 13:58:40.273118 9524 raft_consensus.cc:2468] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 6.
I20260503 13:58:40.275743 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.275774 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.274019 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030fcc1b19c3489ea0ca320576ed54ee" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:40.222446 10842 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.277282 9450 raft_consensus.cc:2393] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:40.277165 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.278868 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:40.279654 9491 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:40.245899 10879 consensus_queue.cc:1048] T a5439e0e393b4112a26459eb56502fae P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
W20260503 13:58:40.319473 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 69190.3us
I20260503 13:58:40.278856 10925 consensus_queue.cc:1059] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.119s
I20260503 13:58:40.273723 10898 raft_consensus.cc:2749] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:40.338325 10891 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
W20260503 13:58:40.395028 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 115435us
I20260503 13:58:40.217384 10639 consensus_queue.cc:237] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.400619 10896 consensus_queue.cc:1059] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260503 13:58:40.401202 9491 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:40.338838 10899 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:40.402210 10899 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:40.402554 10899 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
W20260503 13:58:40.403350 10924 log.cc:927] Time spent T 94b14392ab2e47fea655f24d4c3e5da5 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.229s user 0.000s sys 0.000s
I20260503 13:58:40.403844 10906 raft_consensus.cc:2749] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260503 13:58:40.268678 10649 raft_consensus.cc:2804] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election won for term 5
W20260503 13:58:40.413309 9600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 485) took 1919 ms (client timeout 1935 ms). Trace:
W20260503 13:58:40.413691 9600 rpcz_store.cc:269] 0503 13:58:38.495371 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:38.495878 (+ 507us) service_pool.cc:224] Handling call
0503 13:58:40.413123 (+1917245us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b54003f0cd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:40.413256 (+ 133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2129588096}
W20260503 13:58:40.418908 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:40.421271 9491 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:40.196310 10907 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.243198 10920 raft_consensus.cc:697] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [term 3 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:40.255743 10660 raft_consensus.cc:2749] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:40.339178 9418 leader_election.cc:400] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Vote denied by peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805) with higher term. Message: Invalid argument: T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for earlier term 6. Current term is 7.
I20260503 13:58:40.454952 9418 leader_election.cc:403] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260503 13:58:40.390223 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 141940us
I20260503 13:58:40.389959 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.339941 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 430) took 1778 ms. Trace:
W20260503 13:58:40.456367 9416 leader_election.cc:343] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:40.390453 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46726 (request call id 527) took 1004 ms. Trace:
I20260503 13:58:40.456840 9416 leader_election.cc:304] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:40.457072 9451 rpcz_store.cc:276] 0503 13:58:39.385702 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:39.428779 (+ 43077us) service_pool.cc:224] Handling call
0503 13:58:40.360883 (+932104us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b540016fb54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:40.390395 (+ 29512us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":341057408}
I20260503 13:58:40.389923 9524 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:40.391660 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 490) took 2384 ms. Trace:
I20260503 13:58:40.278412 10193 raft_consensus.cc:515] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.391882 10867 raft_consensus.cc:2749] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:40.360652 10856 raft_consensus.cc:2804] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:40.393427 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.268438 10880 raft_consensus.cc:2749] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:40.471021 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.456300 9597 rpcz_store.cc:276] 0503 13:58:38.561029 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:38.561147 (+ 118us) service_pool.cc:224] Handling call
0503 13:58:39.958342 (+1397195us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:39.958972 (+ 630us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:40.339733 (+380761us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b5400260cd0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:40.339884 (+ 151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3184423808}
W20260503 13:58:40.472455 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 49906.4us
I20260503 13:58:40.473698 9491 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:40.260685 10885 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260503 13:58:40.425360 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.428763 9643 tablet_bootstrap.cc:654] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:40.433301 10912 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.474833 9447 raft_consensus.cc:2393] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:40.459017 9492 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:40.437398 10636 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:40.470872 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 193732us
I20260503 13:58:40.476217 9418 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:40.439924 10638 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.441340 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:40.165066 9405 catalog_manager.cc:6134] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Tablet 96fdbf26d7134a238a95b243fe5b8aac (table test-workload [id=54209d96daad4531bdc33057c81f7430]) was not created within the allowed timeout. Replacing with a new tablet e8904d06ad7a4801a420863e52412455
I20260503 13:58:40.444625 10855 raft_consensus.cc:2804] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election won for term 4
I20260503 13:58:40.447316 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.448206 9519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 513) took 1906 ms. Trace:
I20260503 13:58:40.447933 10869 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:40.448920 10895 log.cc:927] Time spent T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.135s user 0.003s sys 0.006s
I20260503 13:58:40.448989 9444 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:40.545567 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.546590 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.547631 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.394567 10870 raft_consensus.cc:2804] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:40.449874 10852 raft_consensus.cc:2804] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:40.553805 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.554322 9444 raft_consensus.cc:3060] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:40.554795 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.450065 10770 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:40.450735 9593 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 433) took 1888 ms (client timeout 1502 ms). Trace:
I20260503 13:58:40.450861 10802 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.451611 9445 raft_consensus.cc:2410] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260503 13:58:40.452313 10899 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.453750 9521 raft_consensus.cc:2468] T 754ac001a92e4dd6a0b6620803b83722 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:40.454120 10721 raft_consensus.cc:2749] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260503 13:58:40.436676 10768 consensus_meta.cc:287] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.710s user 0.008s sys 0.000s
I20260503 13:58:40.471894 9451 raft_consensus.cc:2376] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4. Re-sending same reply.
I20260503 13:58:40.481567 9452 raft_consensus.cc:3060] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Advancing to term 7
W20260503 13:58:40.528287 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 51349.6us
I20260503 13:58:40.529816 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:40.569192 9418 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:40.569486 9453 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:40.528728 9519 rpcz_store.cc:276] 0503 13:58:38.542133 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:38.849261 (+307128us) service_pool.cc:224] Handling call
0503 13:58:38.861616 (+ 12355us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:40.176143 (+1314527us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:40.448009 (+271866us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540038ed54. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:40.448154 (+ 145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3021879552}
I20260503 13:58:40.481698 9416 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:40.494944 9450 raft_consensus.cc:2468] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:40.570281 9418 leader_election.cc:304] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:40.569133 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.571002 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:40.544587 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 70948.3us
I20260503 13:58:40.526630 10649 raft_consensus.cc:697] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:40.470090 10445 consensus_queue.cc:1048] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.096s
W20260503 13:58:40.556151 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 97684.4us
I20260503 13:58:40.470440 10850 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:40.557482 9526 raft_consensus.cc:1240] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:40.557840 10852 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:40.480672 10885 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:40.477018 10798 raft_consensus.cc:2749] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:40.475646 10856 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:40.475874 10618 raft_consensus.cc:2749] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805) with higher term. Message: Invalid argument: T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for earlier term 6. Current term is 7.
I20260503 13:58:40.457996 9446 rpcz_store.cc:276] 0503 13:58:37.976046 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:38.341175 (+365129us) service_pool.cc:224] Handling call
0503 13:58:38.368974 (+ 27799us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:38.395290 (+ 26316us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:38.401193 (+ 5903us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:38.453200 (+ 52007us) log.cc:844] Serialized 26 byte log entry
0503 13:58:38.583306 (+130106us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:38.605350 (+ 22044us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:38.614513 (+ 9163us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:40.203104 (+1588591us) spinlock_profiling.cc:243] Waited 3.63 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:40.256013 (+ 52909us) raft_consensus.cc:1685] finished
0503 13:58:40.260917 (+ 4904us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:40.360312 (+ 99395us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1514,"raft.queue_time_us":1608,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":3483,"spinlock_wait_cycles":8344832,"thread_start_us":91220,"threads_started":1,"wal-append.queue_time_us":163835}
I20260503 13:58:40.564980 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.481061 10660 raft_consensus.cc:2804] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader election won for term 6
I20260503 13:58:40.566887 9520 raft_consensus.cc:2468] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:40.573778 9521 raft_consensus.cc:1275] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260503 13:58:40.550467 9417 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:40.575503 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:40.571301 9451 raft_consensus.cc:3060] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:40.458652 10847 raft_consensus.cc:2749] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:40.550558 10884 raft_consensus.cc:2749] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260503 13:58:40.689711 10649 consensus_queue.cc:237] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.576884 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:40.578018 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 96379.9us
I20260503 13:58:40.571612 10886 raft_consensus.cc:2749] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:40.562981 9444 raft_consensus.cc:2468] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:40.713722 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:40.714264 9444 raft_consensus.cc:2468] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:40.578576 10635 raft_consensus.cc:2804] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:40.448951 10916 raft_consensus.cc:515] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.565789 10912 raft_consensus.cc:2804] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader election won for term 7
W20260503 13:58:40.640576 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 68009us
I20260503 13:58:40.722661 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.566602 10855 raft_consensus.cc:697] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:40.577665 10867 raft_consensus.cc:2804] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:40.557206 10170 consensus_queue.cc:1059] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 0, Time since last communication: 0.081s
I20260503 13:58:40.557238 10920 consensus_queue.cc:237] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.647962 9492 leader_election.cc:304] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:40.669437 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.733534 9519 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 6.
W20260503 13:58:40.564203 9593 rpcz_store.cc:269] 0503 13:58:38.561913 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:38.586869 (+ 24956us) service_pool.cc:224] Handling call
0503 13:58:40.450545 (+1863676us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400259ed4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:40.450670 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2402121984}
I20260503 13:58:40.576480 10768 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.582398 10771 consensus_queue.cc:1059] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.067s
I20260503 13:58:40.646421 10899 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.644646 10842 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:40.749595 9351 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51388 (request call id 82) took 11885 ms (client timeout 15000 ms). Trace:
I20260503 13:58:40.682042 10898 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:40.682240 10732 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:40.752784 10654 raft_consensus.cc:3055] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:40.753194 10654 raft_consensus.cc:740] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:40.753929 10654 consensus_queue.cc:260] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.754804 10654 raft_consensus.cc:3060] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:40.685747 9449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 524) took 1313 ms. Trace:
I20260503 13:58:40.755494 9449 rpcz_store.cc:276] 0503 13:58:39.372134 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:39.470339 (+ 98205us) service_pool.cc:224] Handling call
0503 13:58:39.470649 (+ 310us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:39.485726 (+ 15077us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:39.485756 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:39.490616 (+ 4860us) log.cc:844] Serialized 26 byte log entry
0503 13:58:40.232797 (+742181us) spinlock_profiling.cc:243] Waited 6.12 ms on lock 0x7b18001248a0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea70587 00007f9f47869bc9 00007f9f3ea8ccdb 00007f9f3ea8af7f 00007f9f3ea8d242 00007f9f416c45ab 00007f9f416ca0c3 00007f9f416ca471 00007f9f41ae6def
0503 13:58:40.338470 (+105673us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:40.338511 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:40.338577 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:40.685480 (+346903us) raft_consensus.cc:1685] finished
0503 13:58:40.685500 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:40.685690 (+ 190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14742,"raft.queue_time_us":31063,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":7460,"spinlock_wait_cycles":14087424,"thread_start_us":75682,"threads_started":1,"wal-append.queue_time_us":1029526}
I20260503 13:58:40.756964 10463 raft_consensus.cc:3055] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 6 LEADER]: Stepping down as leader of term 6
I20260503 13:58:40.757308 10463 raft_consensus.cc:740] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
W20260503 13:58:40.686450 10936 log.cc:927] Time spent T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
I20260503 13:58:40.639695 10850 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.758055 10463 consensus_queue.cc:260] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.758955 10463 raft_consensus.cc:3060] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:40.759629 10850 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:40.692879 10945 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:40.760231 10945 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.760751 9528 raft_consensus.cc:1240] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 6. Current term is 7. Ops: []
I20260503 13:58:40.712297 10638 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.752848 10732 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.694075 10891 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:40.762444 10891 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.762866 10732 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:40.763911 10802 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.764200 10891 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:40.694459 10879 raft_consensus.cc:493] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:40.730171 10867 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:40.764963 10867 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:40.764933 10879 raft_consensus.cc:515] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.765599 10945 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:40.766834 10879 leader_election.cc:290] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:40.696699 10946 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:40.716427 10193 leader_election.cc:290] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:40.716756 10660 raft_consensus.cc:697] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [term 6 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:40.752806 10898 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.769441 10898 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:40.767479 10946 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.730911 10635 raft_consensus.cc:697] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:40.771589 10946 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:40.771947 10635 consensus_queue.cc:237] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.745844 10927 consensus_queue.cc:1048] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20260503 13:58:40.774745 10916 leader_election.cc:290] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.565157 10871 raft_consensus.cc:2804] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:40.701472 9566 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:40.702178 9452 raft_consensus.cc:2468] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 7.
W20260503 13:58:40.749989 9351 rpcz_store.cc:269] 0503 13:58:28.762435 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:28.771669 (+ 9234us) service_pool.cc:224] Handling call
0503 13:58:40.644463 (+11872794us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0503 13:58:34.070504 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 13:58:34.147229 (+ 76725us) write_op.cc:432] Acquiring schema lock in shared mode
0503 13:58:34.147277 (+ 48us) write_op.cc:435] Acquired schema lock
0503 13:58:34.147295 (+ 18us) tablet.cc:662] Decoding operations
0503 13:58:34.151600 (+ 4305us) write_op.cc:620] Acquiring the partition lock for write op
0503 13:58:34.151689 (+ 89us) write_op.cc:641] Partition lock acquired for write op
0503 13:58:34.151723 (+ 34us) tablet.cc:685] Acquiring locks for 16 operations
0503 13:58:34.152875 (+ 1152us) tablet.cc:701] Row locks acquired
0503 13:58:34.152895 (+ 20us) write_op.cc:260] PREPARE: finished
0503 13:58:34.153035 (+ 140us) write_op.cc:270] Start()
0503 13:58:34.153152 (+ 117us) write_op.cc:276] Timestamp: P: 1777816714152980 usec, L: 0
0503 13:58:34.153198 (+ 46us) op_driver.cc:348] REPLICATION: starting
0503 13:58:34.153846 (+ 648us) log.cc:844] Serialized 6309 byte log entry
0503 13:58:34.566897 (+413051us) op_driver.cc:464] REPLICATION: finished
0503 13:58:34.723901 (+157004us) write_op.cc:301] APPLY: starting
0503 13:58:34.723985 (+ 84us) tablet.cc:1367] starting BulkCheckPresence
0503 13:58:34.798511 (+ 74526us) tablet.cc:1370] finished BulkCheckPresence
0503 13:58:34.836104 (+ 37593us) tablet.cc:1372] starting ApplyRowOperation cycle
0503 13:58:34.961020 (+124916us) tablet.cc:1383] finished ApplyRowOperation cycle
0503 13:58:35.004952 (+ 43932us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0503 13:58:35.005013 (+ 61us) write_op.cc:312] APPLY: finished
0503 13:58:35.005744 (+ 731us) log.cc:844] Serialized 112 byte log entry
0503 13:58:35.006502 (+ 758us) write_op.cc:489] Releasing partition, row and schema locks
0503 13:58:35.006650 (+ 148us) write_op.cc:454] Released schema lock
0503 13:58:35.006723 (+ 73us) write_op.cc:341] FINISH: Updating metrics
0503 13:58:35.008758 (+ 2035us) write_op.cc:489] Releasing partition, row and schema locks
0503 13:58:35.008786 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6684,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":138790,"apply.run_cpu_time_us":12353,"apply.run_wall_time_us":283178,"num_ops":16,"prepare.queue_time_us":169989,"prepare.run_cpu_time_us":28480,"prepare.run_wall_time_us":171206,"raft.queue_time_us":174071,"raft.run_cpu_time_us":40215,"raft.run_wall_time_us":125577,"replication_time_us":395017,"thread_start_us":354514,"threads_started":4,"wal-append.queue_time_us":121232,"wal-append.run_cpu_time_us":15159,"wal-append.run_wall_time_us":1726562}]]}
I20260503 13:58:40.750001 10730 raft_consensus.cc:2804] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election won for term 6
I20260503 13:58:40.788645 10870 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:40.789112 10870 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:40.789878 9452 raft_consensus.cc:1275] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260503 13:58:40.708302 9447 raft_consensus.cc:2468] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
W20260503 13:58:40.710244 9405 catalog_manager.cc:6134] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Tablet 94b14392ab2e47fea655f24d4c3e5da5 (table test-workload [id=54209d96daad4531bdc33057c81f7430]) was not created within the allowed timeout. Replacing with a new tablet a420fe6cc288440c8be8654ef9820d54
I20260503 13:58:40.482060 10756 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:40.716616 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:40.791988 9447 raft_consensus.cc:1240] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: [4.4-5.5]
I20260503 13:58:40.718837 9451 raft_consensus.cc:2468] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:40.723827 10907 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.720069 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:40.720675 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:40.801431 9524 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:40.702651 10885 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:40.704367 10856 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
W20260503 13:58:40.733042 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 85151.7us
I20260503 13:58:40.745385 9567 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:40.763356 10170 raft_consensus.cc:3055] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:40.751065 9519 raft_consensus.cc:1275] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 8 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260503 13:58:40.784961 9448 raft_consensus.cc:2468] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 8.
I20260503 13:58:40.745719 10770 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.775290 10660 consensus_queue.cc:237] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.745062 10912 raft_consensus.cc:697] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [term 7 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:40.792038 9446 raft_consensus.cc:2368] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2.
I20260503 13:58:40.802417 9447 raft_consensus.cc:1240] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260503 13:58:40.701373 10852 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
W20260503 13:58:40.793278 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 96698us
I20260503 13:58:40.787015 10871 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:40.823475 9447 raft_consensus.cc:1275] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260503 13:58:40.818979 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 73669.8us
I20260503 13:58:40.799576 10851 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:40.825223 9566 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:40.829523 9566 leader_election.cc:304] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:40.792241 10721 consensus_queue.cc:1048] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.043s
W20260503 13:58:40.832667 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 101977us
I20260503 13:58:40.799652 10927 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:40.833277 9566 leader_election.cc:304] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:40.829530 9492 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:40.800969 10295 raft_consensus.cc:2749] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260503 13:58:40.831717 10869 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.803298 10639 raft_consensus.cc:493] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:40.807559 10170 raft_consensus.cc:740] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:40.816431 10918 raft_consensus.cc:3055] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Stepping down as leader of term 3
I20260503 13:58:40.807057 10730 raft_consensus.cc:697] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [term 6 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:40.846239 9522 raft_consensus.cc:1240] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:40.780939 10798 consensus_queue.cc:1048] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.051s
I20260503 13:58:40.834698 9566 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:40.853576 9524 raft_consensus.cc:2468] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:40.854735 9521 raft_consensus.cc:1275] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260503 13:58:40.854624 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.855460 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.855654 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.860716 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.823964 10756 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260503 13:58:40.832253 10871 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:40.862617 9418 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:40.838515 10850 raft_consensus.cc:3060] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:40.871192 9566 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:40.869359 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:40.840775 10463 raft_consensus.cc:2749] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260503 13:58:40.841449 10849 raft_consensus.cc:2804] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader election won for term 7
I20260503 13:58:40.877388 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.878240 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:40.882515 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.827279 10886 consensus_queue.cc:1059] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260503 13:58:40.855091 10899 consensus_queue.cc:1048] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.030s
I20260503 13:58:40.865866 10756 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:40.841549 10891 raft_consensus.cc:2749] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260503 13:58:40.886229 9566 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:40.896922 10732 raft_consensus.cc:2804] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:40.897158 10170 consensus_queue.cc:260] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.840094 10927 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.848635 10639 raft_consensus.cc:515] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.834726 10851 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.899715 10867 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.903257 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:40.917775 10948 log.cc:927] Time spent T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260503 13:58:40.878473 10544 raft_consensus.cc:2804] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:40.849215 10918 raft_consensus.cc:740] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:40.912451 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:40.914698 10950 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:40.900221 10730 consensus_queue.cc:237] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.900794 10945 consensus_queue.cc:1048] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260503 13:58:40.873170 10842 consensus_queue.cc:1048] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260503 13:58:40.885977 10849 raft_consensus.cc:697] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [term 7 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:40.938620 10849 consensus_queue.cc:237] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.879637 10193 raft_consensus.cc:2749] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260503 13:58:40.920233 9566 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:40.918576 9643 tablet_bootstrap.cc:492] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:40.942077 10544 raft_consensus.cc:697] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:40.942258 9643 ts_tablet_manager.cc:1403] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.763s user 0.010s sys 0.064s
I20260503 13:58:40.919615 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.921475 9519 raft_consensus.cc:1275] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260503 13:58:40.943898 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:40.944703 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 101449us
I20260503 13:58:40.945449 9527 raft_consensus.cc:2410] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260503 13:58:40.946974 9566 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:40.917338 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.864804 10802 consensus_queue.cc:1059] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260503 13:58:40.904240 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:40.847432 10855 consensus_queue.cc:237] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.929049 10639 leader_election.cc:290] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.930624 9446 raft_consensus.cc:1240] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:40.932133 9447 raft_consensus.cc:2393] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:40.955545 9524 raft_consensus.cc:2393] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:40.934296 9452 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:40.919384 10732 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:40.956890 10732 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:40.864612 10618 consensus_queue.cc:1059] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260503 13:58:40.926594 10295 raft_consensus.cc:2804] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:40.936282 10867 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:40.920697 10756 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:40.958653 9447 raft_consensus.cc:1240] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:40.963845 10770 consensus_queue.cc:1059] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260503 13:58:40.964597 10770 raft_consensus.cc:3055] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:40.964933 10770 raft_consensus.cc:740] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:40.965619 10770 consensus_queue.cc:260] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.966658 10770 raft_consensus.cc:3060] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:40.970834 9452 raft_consensus.cc:2468] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:40.972126 10756 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.970669 10732 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.975584 9492 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:40.976495 10869 raft_consensus.cc:2804] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:40.976987 10869 raft_consensus.cc:697] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:40.977773 10869 consensus_queue.cc:237] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:40.986629 9566 leader_election.cc:304] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:40.990376 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d04573f959de406a9590a1603f633e7e" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:40.935321 10170 raft_consensus.cc:3060] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:41.012686 10891 raft_consensus.cc:2804] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260503 13:58:41.013026 9566 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:41.015168 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 48036us
I20260503 13:58:40.984726 10193 raft_consensus.cc:2749] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:40.903910 10912 consensus_queue.cc:237] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.020190 9566 leader_election.cc:400] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599) with higher term. Message: Invalid argument: T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2.
I20260503 13:58:41.020462 9566 leader_election.cc:403] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Cancelling election due to peer responding with higher term
I20260503 13:58:41.009625 10445 raft_consensus.cc:3055] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:40.987870 10295 raft_consensus.cc:697] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:40.990263 10945 raft_consensus.cc:2804] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:41.009902 10654 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:41.033582 10850 raft_consensus.cc:2749] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260503 13:58:41.035346 9568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:41.013119 10891 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:40.990523 10842 consensus_queue.cc:1048] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260503 13:58:41.019383 10913 raft_consensus.cc:2804] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:41.034535 10945 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:41.041455 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.042034 9594 raft_consensus.cc:2468] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:41.029229 10946 raft_consensus.cc:3060] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 2
I20260503 13:58:41.029747 10445 raft_consensus.cc:740] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:41.043792 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.041026 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.044338 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.032179 10733 consensus_queue.cc:1048] T 322e401806f7452c9714ce855e35ca9b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.013s
I20260503 13:58:41.046963 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.047389 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
@ 0x7f9f3ec5da32 testing::Test::Run() at ??:0
I20260503 13:58:41.051921 9491 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:41.052244 10295 consensus_queue.cc:237] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.061962 9602 raft_consensus.cc:2468] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:40.944427 10950 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.036638 10891 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:41.064478 9491 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.040091 10945 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:41.067180 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.068336 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.070498 9601 raft_consensus.cc:3060] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:41.070674 9602 raft_consensus.cc:2393] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 7: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:41.037137 10654 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.072126 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.072903 9595 raft_consensus.cc:2393] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 1: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:41.073638 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.074137 9595 raft_consensus.cc:3060] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:41.076848 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:41.077860 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:41.078400 10870 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.078756 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:41.079715 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:41.079876 9449 raft_consensus.cc:1240] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:41.041746 10846 raft_consensus.cc:493] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:41.082741 9599 raft_consensus.cc:2468] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:41.016876 10544 consensus_queue.cc:237] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.083611 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.043978 10913 raft_consensus.cc:697] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:41.090860 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.091506 9599 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:41.091567 9602 raft_consensus.cc:2368] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 6. Current term is 7.
I20260503 13:58:41.070238 10869 raft_consensus.cc:2804] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:41.092281 10869 raft_consensus.cc:493] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:41.092640 10869 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:41.093668 9566 leader_election.cc:304] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:41.095608 9643 raft_consensus.cc:359] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.096338 9566 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:41.096735 9418 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:41.096853 9405 catalog_manager.cc:6134] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Tablet a5439e0e393b4112a26459eb56502fae (table test-workload [id=54209d96daad4531bdc33057c81f7430]) was not created within the allowed timeout. Replacing with a new tablet 78f9088aa15f4a3f9d8b38bf19977e3b
I20260503 13:58:41.095983 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:41.096266 9643 raft_consensus.cc:385] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:41.097386 9566 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:41.097728 9643 raft_consensus.cc:740] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:41.098412 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:41.099073 9448 raft_consensus.cc:2393] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 7: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:41.098896 9566 leader_election.cc:304] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:41.100979 9566 leader_election.cc:304] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:41.104511 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:41.083164 10896 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:41.071403 10730 raft_consensus.cc:2749] T 54bf4ee675fb4fcf800b317817906dcc P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:41.163231 10463 raft_consensus.cc:2749] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:41.106259 9453 raft_consensus.cc:1275] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 8 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260503 13:58:41.031335 10889 consensus_queue.cc:1059] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260503 13:58:41.106719 9643 consensus_queue.cc:260] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:41.164736 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 63949.6us
I20260503 13:58:41.165540 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:41.100485 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.161983 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.218015 9602 raft_consensus.cc:2393] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 2: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:41.163393 9599 raft_consensus.cc:2468] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:41.196692 9446 raft_consensus.cc:1275] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260503 13:58:41.171222 10896 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.137030 10898 consensus_queue.cc:1059] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.096s
I20260503 13:58:41.219182 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d04573f959de406a9590a1603f633e7e" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.223968 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 540) took 1076 ms. Trace:
I20260503 13:58:41.201936 9643 ts_tablet_manager.cc:1434] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 0.259s user 0.007s sys 0.010s
I20260503 13:58:41.105561 10445 consensus_queue.cc:260] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.205194 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:41.164017 10733 raft_consensus.cc:3060] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:41.209587 10871 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.210284 10927 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:41.211112 10969 log.cc:927] Time spent T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20260503 13:58:41.091130 10846 raft_consensus.cc:515] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:41.233096 10961 log.cc:927] Time spent T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.166s user 0.000s sys 0.000s
I20260503 13:58:41.233170 9446 raft_consensus.cc:1240] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 6. Current term is 7. Ops: []
I20260503 13:58:41.214047 9448 raft_consensus.cc:1275] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260503 13:58:41.234462 9446 raft_consensus.cc:1275] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260503 13:58:41.214707 10870 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:41.234897 10733 raft_consensus.cc:2749] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:41.215961 10885 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.215632 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 542) took 1041 ms. Trace:
I20260503 13:58:41.235671 10639 consensus_queue.cc:1048] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:58:41.236600 10885 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:41.235523 9525 rpcz_store.cc:276] 0503 13:58:40.174358 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:40.215756 (+ 41398us) service_pool.cc:224] Handling call
0503 13:58:40.216929 (+ 1173us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:40.482211 (+265282us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:40.552818 (+ 70607us) log.cc:844] Serialized 14 byte log entry
0503 13:58:40.699977 (+147159us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:40.700032 (+ 55us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:40.700096 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:41.215431 (+515335us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b54003d3bd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:41.215475 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:41.215581 (+ 106us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":74315,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":1309,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":2397,"spinlock_wait_cycles":609569152,"thread_start_us":146493,"threads_started":1,"wal-append.queue_time_us":137020}
I20260503 13:58:41.164372 10850 raft_consensus.cc:2749] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260503 13:58:41.237329 9444 raft_consensus.cc:2393] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 7: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:41.224336 9526 rpcz_store.cc:276] 0503 13:58:40.147637 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:40.212262 (+ 64625us) service_pool.cc:224] Handling call
0503 13:58:40.212518 (+ 256us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:40.582066 (+369548us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:41.223750 (+641684us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b54001554d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:41.223905 (+ 155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":792967424}
W20260503 13:58:41.224699 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260503 13:58:41.238322 10962 log.cc:927] Time spent T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
W20260503 13:58:41.220392 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 45396.6us
I20260503 13:58:41.240579 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:41.241065 9526 raft_consensus.cc:3060] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Advancing to term 7
W20260503 13:58:41.241765 10970 log.cc:927] Time spent T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260503 13:58:41.243114 10852 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.243717 9595 raft_consensus.cc:2468] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:41.229347 10445 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:41.244951 10852 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:41.196877 10950 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:41.229871 10871 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:41.230407 10869 raft_consensus.cc:515] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.234392 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.166565 10618 raft_consensus.cc:2804] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:41.241536 10170 consensus_queue.cc:1059] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260503 13:58:41.229323 10898 raft_consensus.cc:3055] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:41.226938 10913 consensus_queue.cc:237] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:41.247825 9523 raft_consensus.cc:1240] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:41.234823 9446 pending_rounds.cc:85] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6: Aborting all ops after (but not including) 3
I20260503 13:58:41.254609 9526 raft_consensus.cc:2468] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 7.
I20260503 13:58:41.268916 9446 pending_rounds.cc:107] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260503 13:58:41.261232 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.262046 9599 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 6.
I20260503 13:58:41.269181 9446 raft_consensus.cc:2889] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:41.246429 9593 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:41.270987 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:41.262604 9566 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:41.267309 9643 tablet_bootstrap.cc:492] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868: Bootstrap starting.
I20260503 13:58:41.269687 9418 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:41.274523 9566 leader_election.cc:304] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:41.275556 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.276263 9599 raft_consensus.cc:2393] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 6: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:41.341810 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:41.346249 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.346932 9599 raft_consensus.cc:2393] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 6: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
W20260503 13:58:41.345405 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 71039.9us
I20260503 13:58:41.348999 9566 leader_election.cc:304] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:41.345453 10927 raft_consensus.cc:2749] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260503 13:58:41.265686 10618 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:41.353221 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030fcc1b19c3489ea0ca320576ed54ee" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.369475 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:41.355098 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030fcc1b19c3489ea0ca320576ed54ee" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:41.358342 10976 log.cc:927] Time spent T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20260503 13:58:41.358424 9525 raft_consensus.cc:1240] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: []
I20260503 13:58:41.360270 10869 leader_election.cc:290] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:41.378561 9416 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:41.360809 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.267989 10898 raft_consensus.cc:740] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
W20260503 13:58:41.365438 10965 log.cc:927] Time spent T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20260503 13:58:41.345975 10886 raft_consensus.cc:2804] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader election won for term 7
I20260503 13:58:41.256793 10885 consensus_queue.cc:1048] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.090s
I20260503 13:58:41.376369 9491 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:41.382416 9491 leader_election.cc:304] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.383144 9491 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:41.377851 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:41.257100 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.275753 10879 raft_consensus.cc:2749] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:41.371156 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.384876 9599 raft_consensus.cc:2468] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:41.386622 9491 leader_election.cc:400] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Vote denied by peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997) with higher term. Message: Invalid argument: T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 6. Current term is 7.
I20260503 13:58:41.386843 9491 leader_election.cc:403] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260503 13:58:41.387538 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:41.355719 10459 raft_consensus.cc:2749] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260503 13:58:41.390859 9593 raft_consensus.cc:2468] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:41.377964 10618 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:41.395402 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.396029 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.397961 9416 leader_election.cc:304] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:41.399091 9599 raft_consensus.cc:2410] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260503 13:58:41.399013 9491 leader_election.cc:304] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.401625 9491 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:41.376852 10295 consensus_queue.cc:1048] T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260503 13:58:41.391132 10907 raft_consensus.cc:2764] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260503 13:58:41.389350 10849 raft_consensus.cc:2804] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:41.366746 10889 consensus_queue.cc:1059] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.030s
I20260503 13:58:41.400049 10847 raft_consensus.cc:2749] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997) with higher term. Message: Invalid argument: T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 6. Current term is 7.
I20260503 13:58:41.403505 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.380970 10798 consensus_queue.cc:1059] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20260503 13:58:41.389570 10868 raft_consensus.cc:2749] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260503 13:58:41.419615 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.419719 10953 consensus_queue.cc:1059] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:58:41.417490 10871 raft_consensus.cc:2804] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:41.421183 10871 raft_consensus.cc:697] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:41.421648 10756 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:41.384766 10170 raft_consensus.cc:2804] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260503 13:58:41.404965 10544 raft_consensus.cc:2804] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:41.418483 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:41.391750 10886 raft_consensus.cc:697] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [term 7 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:41.420650 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.436175 9594 raft_consensus.cc:2410] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260503 13:58:41.424645 10849 raft_consensus.cc:697] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:41.437800 10849 consensus_queue.cc:237] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.438242 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.438756 9598 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:41.439018 10851 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:41.442898 9405 catalog_manager.cc:6134] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Tablet 8650e2ab26ac47f8a5a6cd4dd6c30d50 (table test-workload [id=54209d96daad4531bdc33057c81f7430]) was not created within the allowed timeout. Replacing with a new tablet 992ba2a583524607be407a3fd9a2c9e6
I20260503 13:58:41.445093 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.403436 10730 raft_consensus.cc:2749] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260503 13:58:41.423915 10544 raft_consensus.cc:493] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:41.446303 10544 raft_consensus.cc:3060] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:41.433879 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:41.422048 10871 consensus_queue.cc:237] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:41.450155 9405 catalog_manager.cc:6134] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Tablet 5b4c6d7380b242c2930533560310a3f4 (table test-workload [id=54209d96daad4531bdc33057c81f7430]) was not created within the allowed timeout. Replacing with a new tablet 8545fb62952f4508aa2418de8a95037d
I20260503 13:58:41.451030 10945 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:41.451382 9405 catalog_manager.cc:6134] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Tablet 754ac001a92e4dd6a0b6620803b83722 (table test-workload [id=54209d96daad4531bdc33057c81f7430]) was not created within the allowed timeout. Replacing with a new tablet 22e8280ba4d1442c9c9d34fd6cd6af98
I20260503 13:58:41.452064 9602 raft_consensus.cc:2368] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5.
I20260503 13:58:41.424250 10170 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:41.454511 10170 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:41.456126 10945 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:41.456543 10896 leader_election.cc:290] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:41.458250 10891 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.459168 9597 raft_consensus.cc:2468] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:41.459517 10960 raft_consensus.cc:3055] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:41.459908 10960 raft_consensus.cc:740] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:41.460376 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.461407 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.462260 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.462846 9597 raft_consensus.cc:2393] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:41.463443 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.463877 9597 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:41.413677 9595 raft_consensus.cc:2393] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 4: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:41.445183 9598 raft_consensus.cc:2468] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
W20260503 13:58:41.412544 10856 consensus_meta.cc:287] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.588s user 0.003s sys 0.003s
I20260503 13:58:41.466040 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:41.415331 9491 leader_election.cc:343] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997): Illegal state: must be running to vote when last-logged opid is not known
I20260503 13:58:41.466593 9491 leader_election.cc:304] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.466671 9598 raft_consensus.cc:2393] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 1: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:41.467132 9594 raft_consensus.cc:2410] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260503 13:58:41.454530 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.468200 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a74decf860264836999f054535dd35ab" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.468811 9598 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:41.477396 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.478368 9446 raft_consensus.cc:1275] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260503 13:58:41.435556 10798 raft_consensus.cc:3055] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Stepping down as leader of term 4
W20260503 13:58:41.479195 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 63937.7us
I20260503 13:58:41.479455 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.482107 9491 leader_election.cc:304] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:41.482220 9602 raft_consensus.cc:3060] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:41.483307 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.483831 9594 raft_consensus.cc:3060] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:41.484432 9491 leader_election.cc:304] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.487617 9491 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.489071 9491 leader_election.cc:304] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.491427 9598 raft_consensus.cc:2468] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:41.487823 10639 raft_consensus.cc:2749] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260503 13:58:41.493327 10730 raft_consensus.cc:3060] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
W20260503 13:58:41.495941 10966 log.cc:927] Time spent T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.331s user 0.006s sys 0.012s
I20260503 13:58:41.481540 10798 raft_consensus.cc:740] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:41.497267 10891 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:41.497321 10798 consensus_queue.cc:260] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.498612 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.475584 10846 leader_election.cc:290] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:41.501600 10170 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.492319 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.502461 9598 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:41.504962 10170 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:41.484622 10649 raft_consensus.cc:2749] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:41.507673 10170 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:41.508409 9491 leader_election.cc:400] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Vote denied by peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997) with higher term. Message: Invalid argument: T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5.
I20260503 13:58:41.508244 10170 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.508739 9491 leader_election.cc:403] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260503 13:58:41.487279 10899 raft_consensus.cc:2804] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election won for term 3
I20260503 13:58:41.510273 9491 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:41.510903 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:41.511548 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 157657us
I20260503 13:58:41.471094 10856 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.514170 9598 raft_consensus.cc:2468] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:41.519107 9594 raft_consensus.cc:2468] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:41.521219 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.521826 9594 raft_consensus.cc:2393] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 7: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:41.521735 9491 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.522812 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.523319 9598 raft_consensus.cc:2393] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 7: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:41.468218 10960 consensus_queue.cc:260] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.525679 9602 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 7.
@ 0x7f9f3ec5ed49 testing::TestInfo::Run() at ??:0
I20260503 13:58:41.533300 10946 raft_consensus.cc:2749] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: Vote denied by peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599) with higher term. Message: Invalid argument: T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 1. Current term is 2.
I20260503 13:58:41.536481 10170 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:41.506158 10639 raft_consensus.cc:2749] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260503 13:58:41.537647 9491 leader_election.cc:304] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
W20260503 13:58:41.542385 9416 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:41.543359 9416 leader_election.cc:304] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:41.511552 10852 raft_consensus.cc:3060] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:41.520473 10886 consensus_queue.cc:237] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.561688 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:41.562783 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:41.524812 10862 raft_consensus.cc:2804] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:41.563664 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:41.564033 10862 raft_consensus.cc:697] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:41.564332 9449 raft_consensus.cc:2393] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:41.564900 10862 consensus_queue.cc:237] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.565975 10654 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:41.573527 9601 raft_consensus.cc:2468] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 7.
I20260503 13:58:41.574862 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.575796 9416 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.576700 9416 leader_election.cc:304] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.545051 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:41.541102 10899 raft_consensus.cc:697] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [term 3 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:41.545735 10649 raft_consensus.cc:2749] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260503 13:58:41.547773 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:41.548328 10544 raft_consensus.cc:515] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.548689 9596 raft_consensus.cc:2468] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:41.549031 10847 raft_consensus.cc:2804] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:41.551327 9597 raft_consensus.cc:2468] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:41.590306 9491 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:41.551864 9643 tablet_bootstrap.cc:654] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868: Neither blocks nor log segments found. Creating new log.
I20260503 13:58:41.554075 10768 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:41.555593 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 559) took 1296 ms. Trace:
I20260503 13:58:41.555998 10730 raft_consensus.cc:2749] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260503 13:58:41.541427 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "754ac001a92e4dd6a0b6620803b83722" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.557593 10618 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.577522 9594 raft_consensus.cc:2393] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:41.584272 9599 raft_consensus.cc:2468] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:41.585357 9416 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.599032 9416 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.585683 10170 raft_consensus.cc:2749] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:41.599574 9601 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 7.
I20260503 13:58:41.599922 10170 raft_consensus.cc:2749] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260503 13:58:41.600389 10170 raft_consensus.cc:2749] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260503 13:58:41.600618 9491 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:41.591534 9453 rpcz_store.cc:276] 0503 13:58:40.259155 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:40.730402 (+471247us) service_pool.cc:224] Handling call
0503 13:58:41.099661 (+369259us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:41.170051 (+ 70390us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:41.555429 (+385378us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b5400303250. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2ac8d 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:41.555541 (+ 112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":848533888}
I20260503 13:58:41.581421 10842 raft_consensus.cc:2749] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260503 13:58:41.601588 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:41.587683 10445 raft_consensus.cc:2749] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260503 13:58:41.600696 9601 raft_consensus.cc:1240] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20260503 13:58:41.597914 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:41.605050 9449 raft_consensus.cc:2393] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 7: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:41.605808 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:41.606400 9449 raft_consensus.cc:2368] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5.
I20260503 13:58:41.602079 9446 raft_consensus.cc:1240] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:41.586920 10985 raft_consensus.cc:2804] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260503 13:58:41.609405 10985 raft_consensus.cc:697] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:41.611469 10985 consensus_queue.cc:237] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.596762 10869 raft_consensus.cc:2804] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election won for term 6
W20260503 13:58:41.616983 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 208008us
I20260503 13:58:41.617354 9444 raft_consensus.cc:2393] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:41.618650 9519 raft_consensus.cc:1240] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
I20260503 13:58:41.601704 9595 raft_consensus.cc:3060] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:41.620939 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:41.622458 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
W20260503 13:58:41.629422 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:41.624293 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:41.604501 10849 consensus_queue.cc:1059] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.012s
I20260503 13:58:41.626844 10869 raft_consensus.cc:697] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 6 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:41.626075 10899 consensus_queue.cc:237] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.628859 10856 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:41.615291 9601 raft_consensus.cc:1275] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:41.604806 10847 raft_consensus.cc:697] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:41.638746 10847 consensus_queue.cc:237] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.621266 10768 raft_consensus.cc:2804] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader election won for term 7
I20260503 13:58:41.601119 10907 consensus_queue.cc:1048] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20260503 13:58:41.643095 10869 consensus_queue.cc:237] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.644078 10918 consensus_queue.cc:260] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.645213 10918 raft_consensus.cc:3060] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:41.646449 9522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 593) took 1253 ms. Trace:
I20260503 13:58:41.646780 9522 rpcz_store.cc:276] 0503 13:58:40.392901 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:40.449203 (+ 56302us) service_pool.cc:224] Handling call
0503 13:58:40.449474 (+ 271us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:40.857320 (+407846us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:41.646263 (+788943us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b540027b5d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:41.646399 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":912035712}
I20260503 13:58:41.647787 10875 raft_consensus.cc:3055] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Stepping down as leader of term 1
I20260503 13:58:41.648115 10875 raft_consensus.cc:740] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:41.641484 10884 consensus_queue.cc:1059] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260503 13:58:41.649890 10561 raft_consensus.cc:3055] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:41.650245 10561 raft_consensus.cc:740] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:41.651156 10852 raft_consensus.cc:2749] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997) with higher term. Message: Invalid argument: T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5.
I20260503 13:58:41.650947 10561 consensus_queue.cc:260] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.651724 10618 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:41.652078 10561 raft_consensus.cc:3060] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:41.653275 10544 leader_election.cc:290] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:41.655957 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:41.660439 10768 raft_consensus.cc:697] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [term 7 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:41.661335 10768 consensus_queue.cc:237] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.662220 10732 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:41.662573 9595 raft_consensus.cc:2468] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:41.664187 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 562) took 1404 ms. Trace:
I20260503 13:58:41.670378 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:41.666453 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:41.671144 9601 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:41.664506 9445 rpcz_store.cc:276] 0503 13:58:40.259789 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:40.754503 (+494714us) service_pool.cc:224] Handling call
0503 13:58:40.754778 (+ 275us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:40.754937 (+ 159us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:40.839713 (+ 84776us) log.cc:844] Serialized 14 byte log entry
0503 13:58:41.663564 (+823851us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x7b1800363940. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea01e1a 00007f9f480a35c9 00007f9f3e8c01f0 00007f9f3ea8ab24 00007f9f3ea8aa95 00007f9f4552e837 00007f9f41b35d87 00007f9f41b4549f 00007f9f41b45418
0503 13:58:41.663761 (+ 197us) log.cc:844] Serialized 14 byte log entry
0503 13:58:41.663916 (+ 155us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:41.663963 (+ 47us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:41.663998 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:41.664051 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:41.664139 (+ 88us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37049,"prepare.queue_time_us":784473,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":3553,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":2420,"spinlock_wait_cycles":5916032,"thread_start_us":131349,"threads_started":2,"wal-append.queue_time_us":192456}
W20260503 13:58:41.664002 10974 log.cc:927] Time spent T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.420s user 0.003s sys 0.000s
I20260503 13:58:41.683765 9522 raft_consensus.cc:1275] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260503 13:58:41.697371 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:41.699291 9445 raft_consensus.cc:3060] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
W20260503 13:58:41.736934 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 137965us
I20260503 13:58:41.737337 9601 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:41.666837 10868 consensus_queue.cc:1059] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260503 13:58:41.739461 9452 raft_consensus.cc:1275] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260503 13:58:41.741720 9491 leader_election.cc:304] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:41.743964 9416 leader_election.cc:304] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.672124 10907 consensus_queue.cc:1048] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260503 13:58:41.744659 9452 raft_consensus.cc:1275] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260503 13:58:41.745149 9416 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:41.738168 10961 log.cc:927] Time spent T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260503 13:58:41.747407 9416 leader_election.cc:304] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.748831 9416 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:41.748430 9452 raft_consensus.cc:1275] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:41.751897 9452 raft_consensus.cc:1240] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 6. Current term is 7. Ops: []
I20260503 13:58:41.752206 9416 leader_election.cc:304] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:41.750355 10987 raft_consensus.cc:2749] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260503 13:58:41.755298 9602 raft_consensus.cc:3060] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:41.749302 10618 raft_consensus.cc:2749] T 3500374093e84b968ed96ed4dbdfda69 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260503 13:58:41.756716 9416 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:41.752661 10650 consensus_queue.cc:1048] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.007s
I20260503 13:58:41.759897 10875 consensus_queue.cc:260] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.761258 10875 raft_consensus.cc:3060] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
W20260503 13:58:41.765334 10995 log.cc:927] Time spent T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.092s user 0.001s sys 0.000s
I20260503 13:58:41.746305 10802 raft_consensus.cc:3060] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:41.709380 10544 consensus_queue.cc:1048] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260503 13:58:41.751595 10852 raft_consensus.cc:2804] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election won for term 6
I20260503 13:58:41.752543 10721 raft_consensus.cc:2749] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:41.786494 9522 raft_consensus.cc:1275] T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
@ 0x7f9f3ec5fd25 testing::TestSuite::Run() at ??:0
I20260503 13:58:41.815840 9447 raft_consensus.cc:1275] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:41.818373 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32fd983d73354e649ff91e64f251e44f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:41.819383 10989 log.cc:927] Time spent T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20260503 13:58:41.825994 10990 log.cc:927] Time spent T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260503 13:58:41.751334 10849 consensus_queue.cc:1048] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.036s
I20260503 13:58:41.784438 10851 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:41.783888 10993 consensus_queue.cc:1048] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260503 13:58:41.776183 10884 raft_consensus.cc:2804] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election won for term 5
W20260503 13:58:41.860221 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 72773.6us
I20260503 13:58:41.868785 9450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 556) took 1610 ms. Trace:
I20260503 13:58:41.864753 10721 consensus_queue.cc:1048] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:58:41.778245 10650 consensus_queue.cc:1059] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260503 13:58:41.892329 9528 raft_consensus.cc:1240] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [term 6 LEADER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:41.894237 9524 raft_consensus.cc:1240] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
W20260503 13:58:41.880005 10988 log.cc:927] Time spent T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.206s user 0.001s sys 0.000s
I20260503 13:58:41.873353 10852 raft_consensus.cc:697] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [term 6 LEADER]: Becoming Leader. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
W20260503 13:58:41.891983 10992 log.cc:927] Time spent T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.181s user 0.003s sys 0.001s
I20260503 13:58:41.873859 9445 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:41.876907 9450 rpcz_store.cc:276] 0503 13:58:40.258522 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:40.578391 (+319869us) service_pool.cc:224] Handling call
0503 13:58:40.581338 (+ 2947us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:40.737351 (+156013us) raft_consensus.cc:1534] Early marking committed up to index 2
0503 13:58:40.786761 (+ 49410us) log.cc:844] Serialized 14 byte log entry
0503 13:58:40.867790 (+ 81029us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:40.867845 (+ 55us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:40.867884 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:41.868527 (+1000643us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54003705d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:41.868603 (+ 76us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:41.868730 (+ 127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35765,"prepare.queue_time_us":53891,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":19789,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":9771,"spinlock_wait_cycles":358327168,"thread_start_us":61323,"threads_started":1,"wal-append.queue_time_us":131163}
I20260503 13:58:41.917181 9450 raft_consensus.cc:1275] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260503 13:58:41.893422 10884 raft_consensus.cc:697] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:41.880271 10851 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.914026 10847 consensus_queue.cc:1048] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260503 13:58:41.960140 9519 raft_consensus.cc:2468] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:41.949442 10918 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:41.962582 9447 raft_consensus.cc:1275] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:41.963145 9418 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:41.950902 10871 consensus_queue.cc:1048] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.012s
I20260503 13:58:41.979708 9643 tablet_bootstrap.cc:492] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868: No bootstrap required, opened a new log
I20260503 13:58:41.980221 9643 ts_tablet_manager.cc:1403] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868: Time spent bootstrapping tablet: real 0.718s user 0.006s sys 0.013s
I20260503 13:58:41.977991 10884 consensus_queue.cc:237] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:41.994156 10802 raft_consensus.cc:2749] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:41.965830 10918 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.020917 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:42.021795 10798 raft_consensus.cc:3060] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:41.995869 10852 consensus_queue.cc:237] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:41.996310 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:41.968839 10998 raft_consensus.cc:2804] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:41.995621 10851 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:42.054263 9449 raft_consensus.cc:1240] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: []
W20260503 13:58:42.035629 11004 log.cc:927] Time spent T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.072s user 0.000s sys 0.002s
W20260503 13:58:42.054165 11010 log.cc:927] Time spent T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.063s user 0.002s sys 0.004s
W20260503 13:58:42.052208 10974 log.cc:927] Time spent T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.060s user 0.000s sys 0.005s
W20260503 13:58:42.067085 10994 log.cc:927] Time spent T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.308s user 0.002s sys 0.000s
W20260503 13:58:42.067795 10967 log.cc:927] Time spent T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.835s user 0.002s sys 0.004s
W20260503 13:58:42.068121 11008 log.cc:927] Time spent T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.064s user 0.002s sys 0.005s
I20260503 13:58:42.072875 10907 consensus_queue.cc:1059] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260503 13:58:42.073017 10756 raft_consensus.cc:3055] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 6 LEADER]: Stepping down as leader of term 6
W20260503 13:58:42.078604 9566 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:42.081100 10849 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:42.052179 10802 consensus_queue.cc:1059] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260503 13:58:42.081593 10849 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.082228 10802 consensus_queue.cc:1243] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer 428b342dc4f041df956015e7d7cd173a log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260503 13:58:42.051728 10733 consensus_queue.cc:1048] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.022s
I20260503 13:58:42.082659 10802 raft_consensus.cc:3055] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:42.082989 10802 raft_consensus.cc:740] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:42.083441 10849 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:42.083698 10802 consensus_queue.cc:260] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.079200 10871 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:42.081617 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:42.085738 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 501) took 1041 ms. Trace:
I20260503 13:58:42.085193 10871 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.086082 9597 rpcz_store.cc:276] 0503 13:58:41.043944 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.592847 (+548903us) service_pool.cc:224] Handling call
0503 13:58:41.593138 (+ 291us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:41.593331 (+ 193us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:41.616535 (+ 23204us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.085448 (+468913us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:42.085494 (+ 46us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:42.085530 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.085590 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:42.085687 (+ 97us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":419864,"prepare.queue_time_us":41696,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":420861,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":75,"thread_start_us":48428,"threads_started":1,"wal-append.queue_time_us":133395}
I20260503 13:58:42.085161 9521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 552) took 1161 ms. Trace:
I20260503 13:58:42.081840 10918 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:42.087550 9521 rpcz_store.cc:276] 0503 13:58:40.923334 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:40.950417 (+ 27083us) service_pool.cc:224] Handling call
0503 13:58:40.950652 (+ 235us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:40.950949 (+ 297us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:40.950978 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:40.951347 (+ 369us) log.cc:844] Serialized 26 byte log entry
0503 13:58:41.035151 (+ 83804us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:41.035188 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:41.036138 (+ 950us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:41.993352 (+957214us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:42.084882 (+ 91530us) raft_consensus.cc:1685] finished
0503 13:58:42.084904 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:42.085100 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":65,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":139,"spinlock_wait_cycles":41082112,"thread_start_us":83422,"threads_started":1,"wal-append.queue_time_us":132011}
W20260503 13:58:42.131469 10974 log.cc:927] Time spent T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20260503 13:58:42.135917 11017 log.cc:927] Time spent T 010e981b4bda450e9e11c3f6ff93f7eb P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20260503 13:58:42.138142 9416 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:42.069322 10998 raft_consensus.cc:697] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
W20260503 13:58:42.142190 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:42.151306 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 53761us
W20260503 13:58:42.154372 9418 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:42.167466 9597 raft_consensus.cc:1240] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:42.168154 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 526) took 1117 ms. Trace:
I20260503 13:58:42.168643 9597 rpcz_store.cc:276] 0503 13:58:41.051094 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.166841 (+1115747us) service_pool.cc:224] Handling call
0503 13:58:42.167311 (+ 470us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:42.167895 (+ 584us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.168098 (+ 203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:42.169973 9597 raft_consensus.cc:1275] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260503 13:58:42.170538 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 527) took 1119 ms. Trace:
I20260503 13:58:42.170863 9597 rpcz_store.cc:276] 0503 13:58:41.051307 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.169400 (+1118093us) service_pool.cc:224] Handling call
0503 13:58:42.169666 (+ 266us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:42.170331 (+ 665us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.170492 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:42.172099 9597 raft_consensus.cc:1275] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260503 13:58:42.172664 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 469) took 1116 ms. Trace:
I20260503 13:58:42.173003 9597 rpcz_store.cc:276] 0503 13:58:41.056117 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.171577 (+1115460us) service_pool.cc:224] Handling call
0503 13:58:42.171821 (+ 244us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:42.172444 (+ 623us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.172619 (+ 175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:42.174089 9597 raft_consensus.cc:1240] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:42.174631 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 470) took 1118 ms. Trace:
I20260503 13:58:42.174958 9597 rpcz_store.cc:276] 0503 13:58:41.056468 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.173686 (+1117218us) service_pool.cc:224] Handling call
0503 13:58:42.173926 (+ 240us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:42.174437 (+ 511us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.174588 (+ 151us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:42.080008 10756 raft_consensus.cc:740] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:42.164094 10802 raft_consensus.cc:3060] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
W20260503 13:58:42.190213 9405 catalog_manager.cc:6134] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Tablet d04573f959de406a9590a1603f633e7e (table test-workload [id=54209d96daad4531bdc33057c81f7430]) was not created within the allowed timeout. Replacing with a new tablet d15bfe564ae84e32a50dae465ec0d8fd
I20260503 13:58:42.187215 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:42.193221 9447 raft_consensus.cc:1275] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260503 13:58:42.186375 10998 consensus_queue.cc:237] T 57b1804f75444731b8b3b6af3c038cf6 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.200402 10946 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:42.200865 10946 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.201403 10913 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:42.201810 10913 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.202656 9567 leader_election.cc:304] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:42.203771 9567 leader_election.cc:304] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:42.201108 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 505) took 1156 ms. Trace:
I20260503 13:58:42.204051 9453 raft_consensus.cc:2410] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260503 13:58:42.177860 10889 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:42.204620 9599 rpcz_store.cc:276] 0503 13:58:41.044767 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.598587 (+553820us) service_pool.cc:224] Handling call
0503 13:58:41.598824 (+ 237us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:41.598998 (+ 174us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:41.702777 (+103779us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.139566 (+436789us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.200827 (+ 61261us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:42.200873 (+ 46us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:42.200907 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.200964 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:42.201056 (+ 92us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":441026,"prepare.queue_time_us":66384,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":72,"raft.queue_time_us":448059,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"spinlock_wait_cycles":7579392,"thread_start_us":143171,"threads_started":2,"wal-append.queue_time_us":173802}
I20260503 13:58:42.202615 10946 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:42.196667 10650 consensus_queue.cc:1048] T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260503 13:58:42.229161 9643 raft_consensus.cc:359] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.229394 9491 leader_election.cc:304] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:42.229789 9643 raft_consensus.cc:385] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260503 13:58:42.230098 9643 raft_consensus.cc:740] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Initialized, Role: FOLLOWER
I20260503 13:58:42.226900 9522 raft_consensus.cc:1275] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260503 13:58:42.203552 10913 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
@ 0x7f9f3ec74815 testing::internal::UnitTestImpl::RunAllTests() at ??:0
W20260503 13:58:42.459297 9405 catalog_manager.cc:6134] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Tablet 5c55d864e0b0416ebdc09ce6e0310a8f (table test-workload [id=54209d96daad4531bdc33057c81f7430]) was not created within the allowed timeout. Replacing with a new tablet 48b36ec7b3cb4cacb44ac10e6ed36652
W20260503 13:58:42.419555 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 184352us
I20260503 13:58:42.457136 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 520) took 1407 ms. Trace:
W20260503 13:58:42.520102 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 316384us
I20260503 13:58:42.274633 9599 raft_consensus.cc:1240] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260503 13:58:42.521257 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 291904us
I20260503 13:58:42.522013 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 479) took 1244 ms. Trace:
I20260503 13:58:42.522377 9599 rpcz_store.cc:276] 0503 13:58:41.277425 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.225216 (+947791us) service_pool.cc:224] Handling call
0503 13:58:42.274408 (+ 49192us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:42.521774 (+247366us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.521961 (+ 187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:42.403693 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 583) took 1338 ms. Trace:
I20260503 13:58:42.523979 9448 rpcz_store.cc:276] 0503 13:58:41.064935 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.234131 (+169196us) service_pool.cc:224] Handling call
0503 13:58:41.234441 (+ 310us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:41.234799 (+ 358us) raft_consensus.cc:1534] Early marking committed up to index 1
0503 13:58:41.743591 (+508792us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.029540 (+285949us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:42.030052 (+ 512us) log.cc:844] Serialized 52 byte log entry
0503 13:58:42.030180 (+ 128us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:42.031242 (+ 1062us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.051354 (+ 20112us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.051434 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:42.403486 (+352052us) raft_consensus.cc:1685] finished
0503 13:58:42.403508 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:42.403633 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":514741,"prepare.queue_time_us":516020,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":64,"raft.queue_time_us":47267,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":14611,"thread_start_us":238767,"threads_started":1,"wal-append.queue_time_us":321397}
I20260503 13:58:42.224458 10459 raft_consensus.cc:493] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:42.535511 10993 consensus_queue.cc:1059] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260503 13:58:42.535562 10856 consensus_queue.cc:1048] T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260503 13:58:42.536666 10993 raft_consensus.cc:3055] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:42.537026 10993 raft_consensus.cc:740] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:42.537735 10993 consensus_queue.cc:260] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.538810 10993 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Advancing to term 6
W20260503 13:58:42.453794 11020 log.cc:927] Time spent T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.250s user 0.000s sys 0.000s
I20260503 13:58:42.228047 10721 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:42.546722 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:42.547772 9448 raft_consensus.cc:2393] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 7: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
W20260503 13:58:42.453042 11009 log.cc:927] Time spent T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.155s user 0.000s sys 0.007s
I20260503 13:58:42.515602 9598 rpcz_store.cc:276] 0503 13:58:41.049798 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.873276 (+823478us) service_pool.cc:224] Handling call
0503 13:58:41.873559 (+ 283us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:41.873740 (+ 181us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:41.892416 (+ 18676us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.223494 (+331078us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.456416 (+232922us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:42.456465 (+ 49us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:42.456505 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.456965 (+ 460us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:42.457083 (+ 118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":445597,"prepare.queue_time_us":33358,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":2600,"raft.queue_time_us":451492,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":71,"spinlock_wait_cycles":4264064,"thread_start_us":116403,"threads_started":2,"wal-append.queue_time_us":73612}
I20260503 13:58:42.462743 10906 raft_consensus.cc:3055] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 6 LEADER]: Stepping down as leader of term 6
I20260503 13:58:42.457648 10884 consensus_queue.cc:1048] T 4d8e0a262a0c48d5b561d53f59fb808a P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.096s
I20260503 13:58:42.553028 10906 raft_consensus.cc:740] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:42.555087 10906 consensus_queue.cc:260] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.555968 10906 raft_consensus.cc:3060] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:42.443105 10798 raft_consensus.cc:493] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:42.556833 10798 raft_consensus.cc:515] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.557356 10770 raft_consensus.cc:3055] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Stepping down as leader of term 2
I20260503 13:58:42.557680 10770 raft_consensus.cc:740] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:42.558355 10770 consensus_queue.cc:260] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.559222 10770 raft_consensus.cc:3060] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:42.434257 10987 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:42.559831 10987 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.336092 10849 raft_consensus.cc:2749] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260503 13:58:42.235096 10756 consensus_queue.cc:260] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.273119 10639 consensus_queue.cc:1048] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.290s
I20260503 13:58:42.563171 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 566) took 1867 ms. Trace:
I20260503 13:58:42.563486 9451 rpcz_store.cc:276] 0503 13:58:40.695541 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:40.800627 (+105086us) service_pool.cc:224] Handling call
0503 13:58:40.801953 (+ 1326us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:40.821796 (+ 19843us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:40.821829 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:41.766082 (+944253us) spinlock_profiling.cc:243] Waited 5.08 ms on lock 0x7b1800363dc0. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea5bbe6 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f40425c81 00007f9f404355e2 00007f9f41b66de1 00007f9f41b2b81a
0503 13:58:41.975802 (+209720us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x7b2c00107268. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f40425c81 00007f9f404355e2 00007f9f41b66de1 00007f9f41b2b81a 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22
0503 13:58:41.981559 (+ 5757us) log.cc:844] Serialized 26 byte log entry
0503 13:58:42.065006 (+ 83447us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:42.069249 (+ 4243us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.069586 (+ 337us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.069667 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:42.562855 (+493188us) raft_consensus.cc:1685] finished
0503 13:58:42.562873 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:42.563081 (+ 208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38538,"prepare.queue_time_us":5706,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":3397,"raft.queue_time_us":22976,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":5989,"spinlock_wait_cycles":18369536,"thread_start_us":44829,"threads_started":1,"wal-append.queue_time_us":118957}
I20260503 13:58:42.564757 11031 raft_consensus.cc:493] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:42.565276 11031 raft_consensus.cc:515] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:42.566838 11025 log.cc:927] Time spent T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.132s user 0.001s sys 0.000s
I20260503 13:58:42.488683 10869 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:42.549160 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:42.233130 10920 raft_consensus.cc:2749] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:42.549975 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:42.550756 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:42.576987 9524 raft_consensus.cc:2393] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 7: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:42.549397 10856 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:42.453899 10871 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:42.456671 10953 raft_consensus.cc:493] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:42.456806 10463 raft_consensus.cc:493] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:42.541204 10459 raft_consensus.cc:515] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.273182 10918 consensus_queue.cc:1059] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.277s
I20260503 13:58:42.382462 10945 raft_consensus.cc:2749] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:42.464484 10771 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:42.442886 10913 consensus_queue.cc:1048] T ec9b187575714b4884a75ff6aa911e15 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.109s
W20260503 13:58:42.551971 11024 log.cc:927] Time spent T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
I20260503 13:58:42.293798 10925 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:42.228669 10889 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:42.572723 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:42.544981 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:42.437315 10660 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:42.526558 10870 raft_consensus.cc:493] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:42.632606 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:42.578312 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:42.234622 10445 consensus_queue.cc:1048] T c6b3066cc4e4411f91d1cde17a952170 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.315s
I20260503 13:58:42.582531 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b48fff133f84ff7827806460a23803b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:42.638164 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:42.563498 10721 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.640523 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:42.643824 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260503 13:58:42.582826 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 376860us
I20260503 13:58:42.586331 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 588) took 1068 ms. Trace:
I20260503 13:58:42.588869 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:42.628692 9602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 500) took 1585 ms. Trace:
I20260503 13:58:42.645475 9602 rpcz_store.cc:276] 0503 13:58:41.043706 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.526339 (+482633us) service_pool.cc:224] Handling call
0503 13:58:41.755013 (+228674us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b5800010898. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3ea506c1 00007f9f3ea50572 00007f9f4790d1d9 00007f9f46a885e0 00007f9f46b5176b 00007f9f46b517f1 00007f9f46b13680 00007f9f46afdfe9 00007f9f46afda92 00007f9f40067d34
0503 13:58:41.755168 (+ 155us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:42.152332 (+397164us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:42.152363 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:42.152830 (+ 467us) log.cc:844] Serialized 26 byte log entry
0503 13:58:42.222724 (+ 69894us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:42.222763 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.222845 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:42.591935 (+369090us) raft_consensus.cc:1685] finished
0503 13:58:42.591957 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:42.628628 (+ 36671us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":362597,"raft.queue_time_us":375018,"spinlock_wait_cycles":43263104,"thread_start_us":31686,"threads_started":1,"wal-append.queue_time_us":391513}
I20260503 13:58:42.629313 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:42.644891 9453 rpcz_store.cc:276] 0503 13:58:41.518275 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.601381 (+ 83106us) service_pool.cc:224] Handling call
0503 13:58:42.586173 (+984792us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b540029f254. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:42.586283 (+ 110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1380487936}
I20260503 13:58:42.649870 9453 raft_consensus.cc:1275] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260503 13:58:42.631531 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:42.577548 9522 raft_consensus.cc:2393] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 5: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:42.636211 9416 leader_election.cc:304] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:42.645138 9566 leader_election.cc:400] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Vote denied by peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599) with higher term. Message: Invalid argument: T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5.
I20260503 13:58:42.651506 9566 leader_election.cc:403] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260503 13:58:42.652024 9528 raft_consensus.cc:2410] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 3 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260503 13:58:42.652951 9526 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 6.
I20260503 13:58:42.653209 9416 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:42.654932 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:42.658151 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:42.661643 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 111990us
I20260503 13:58:42.662477 9526 raft_consensus.cc:1275] T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:42.578997 10869 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.671603 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:42.672197 9602 raft_consensus.cc:2468] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:42.583968 10856 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.680142 9567 leader_election.cc:304] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:42.681128 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:42.588543 10463 raft_consensus.cc:515] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:42.683559 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:42.668534 11025 log.cc:927] Time spent T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20260503 13:58:42.684523 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:42.657080 10846 raft_consensus.cc:2749] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599) with higher term. Message: Invalid argument: T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5.
I20260503 13:58:42.686003 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:42.594997 10771 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.702732 9599 raft_consensus.cc:2393] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:42.663537 10618 raft_consensus.cc:493] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:42.702821 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:42.705873 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:42.589082 10953 raft_consensus.cc:515] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:42.708101 11022 log.cc:927] Time spent T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.270s user 0.005s sys 0.004s
I20260503 13:58:42.654589 10884 consensus_queue.cc:1059] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260503 13:58:42.706247 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:42.639842 10870 raft_consensus.cc:515] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.656625 10660 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.711706 9602 raft_consensus.cc:2410] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260503 13:58:42.681998 10798 leader_election.cc:290] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:42.636831 10925 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.723483 9491 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:42.701805 10636 raft_consensus.cc:493] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:42.716054 9521 raft_consensus.cc:2468] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:42.768433 9599 raft_consensus.cc:1240] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260503 13:58:42.769043 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 489) took 1394 ms. Trace:
I20260503 13:58:42.769326 9599 rpcz_store.cc:276] 0503 13:58:41.374428 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.717020 (+1342592us) service_pool.cc:224] Handling call
0503 13:58:42.768226 (+ 51206us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:42.768800 (+ 574us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.768992 (+ 192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:42.682186 10768 consensus_queue.cc:1048] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260503 13:58:42.684194 10987 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:42.684110 11031 leader_election.cc:290] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:42.757882 10945 raft_consensus.cc:2749] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260503 13:58:42.755159 10993 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:42.767652 10899 raft_consensus.cc:2749] T 2c918e4ac84c4238a06220bc2817286f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:42.768019 10918 raft_consensus.cc:2749] T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260503 13:58:42.819975 11025 log.cc:927] Time spent T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260503 13:58:42.768190 10884 consensus_queue.cc:1243] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Peer 428b342dc4f041df956015e7d7cd173a log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260503 13:58:42.866091 9453 raft_consensus.cc:1275] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:58:42.866714 9599 raft_consensus.cc:1240] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 1. Current term is 2. Ops: []
W20260503 13:58:42.886994 11028 log.cc:927] Time spent T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.249s user 0.000s sys 0.002s
W20260503 13:58:42.888166 11033 log.cc:927] Time spent T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.251s user 0.000s sys 0.000s
I20260503 13:58:42.712795 10618 raft_consensus.cc:515] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.763628 10636 raft_consensus.cc:515] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:42.912640 9492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:42.923579 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 478) took 1650 ms. Trace:
I20260503 13:58:42.970274 9597 rpcz_store.cc:276] 0503 13:58:41.273192 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.175657 (+902465us) service_pool.cc:224] Handling call
0503 13:58:42.175948 (+ 291us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:42.207032 (+ 31084us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:42.207061 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:42.223049 (+ 15988us) log.cc:844] Serialized 52 byte log entry
0503 13:58:42.464239 (+241190us) spinlock_profiling.cc:243] Waited 58.4 ms on lock 0x7b48001d01d8. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f46a631a1 00007f9f46a5a237 00007f9f46a5a0ca 00007f9f46b5192f 00007f9f46b515f7 00007f9f46b65c1c 00007f9f46b65bc8 00007f9f46b65b48 00007f9f46b65b03 00007f9f46b64dac 00007f9f45a8db36
0503 13:58:42.578399 (+114160us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:42.592762 (+ 14363us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.593136 (+ 374us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.593347 (+ 211us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.593412 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:42.923353 (+329941us) raft_consensus.cc:1685] finished
0503 13:58:42.923374 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:42.923518 (+ 144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46984,"prepare.queue_time_us":100770,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":83,"raft.queue_time_us":56173,"raft.run_cpu_time_us":1891,"raft.run_wall_time_us":267330,"spinlock_wait_cycles":138033792,"thread_start_us":175983,"threads_started":1,"wal-append.queue_time_us":415195}
I20260503 13:58:42.925057 10771 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:42.926298 10463 leader_election.cc:290] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:42.930919 10639 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
W20260503 13:58:42.985248 9494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:42.985029 10639 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:42.987013 10639 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:42.777130 10798 raft_consensus.cc:2749] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260503 13:58:42.987550 9447 raft_consensus.cc:1275] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260503 13:58:42.943423 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 496) took 1567 ms. Trace:
I20260503 13:58:42.988096 9599 rpcz_store.cc:276] 0503 13:58:41.375952 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.769987 (+1394035us) service_pool.cc:224] Handling call
0503 13:58:42.866542 (+ 96555us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:42.943177 (+ 76635us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.943369 (+ 192us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:42.910513 11030 log.cc:927] Time spent T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.179s user 0.000s sys 0.000s
I20260503 13:58:42.998054 9447 raft_consensus.cc:1240] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 2. Current term is 3. Ops: []
I20260503 13:58:42.999833 9597 raft_consensus.cc:1275] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260503 13:58:43.000438 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 530) took 1609 ms. Trace:
I20260503 13:58:43.000751 9597 rpcz_store.cc:276] 0503 13:58:41.390737 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.989079 (+1598342us) service_pool.cc:224] Handling call
0503 13:58:42.989361 (+ 282us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:43.000208 (+ 10847us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.000389 (+ 181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:43.002944 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:43.003677 9444 raft_consensus.cc:2393] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 5: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:43.004516 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:43.004886 10870 leader_election.cc:290] T 11b8ecb173e84681b8fb389c89395838 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:43.005097 9444 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 7.
I20260503 13:58:43.005903 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:43.006423 9444 raft_consensus.cc:2468] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:43.010008 9492 leader_election.cc:304] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:43.013651 9492 leader_election.cc:304] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:43.014851 10768 consensus_queue.cc:1048] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:58:43.015094 9595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 513) took 1966 ms. Trace:
I20260503 13:58:43.015614 10889 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:43.016587 10953 leader_election.cc:290] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:43.017515 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 600) took 1054 ms. Trace:
I20260503 13:58:43.018147 9599 raft_consensus.cc:1275] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260503 13:58:43.018786 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 531) took 1622 ms. Trace:
I20260503 13:58:43.017868 9446 rpcz_store.cc:276] 0503 13:58:41.963812 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.994929 (+ 31117us) service_pool.cc:224] Handling call
0503 13:58:41.995234 (+ 305us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:41.995573 (+ 339us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:41.995600 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:42.034635 (+ 39035us) log.cc:844] Serialized 52 byte log entry
0503 13:58:42.178082 (+143447us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:42.178121 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.178180 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:42.564472 (+386292us) spinlock_profiling.cc:243] Waited 89.5 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:43.017234 (+452762us) raft_consensus.cc:1685] finished
0503 13:58:43.017256 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:43.017462 (+ 206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9473,"raft.queue_time_us":1753,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"spinlock_wait_cycles":205948160,"thread_start_us":105727,"threads_started":1,"wal-append.queue_time_us":162043}
I20260503 13:58:43.019121 9599 rpcz_store.cc:276] 0503 13:58:41.397046 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.993218 (+1596172us) service_pool.cc:224] Handling call
0503 13:58:43.013092 (+ 19874us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:43.018573 (+ 5481us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.018739 (+ 166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:42.981174 10906 raft_consensus.cc:493] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:42.939257 10993 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.028250 9599 raft_consensus.cc:1275] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260503 13:58:43.002974 10618 leader_election.cc:290] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:43.032372 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 535) took 1477 ms. Trace:
I20260503 13:58:43.034762 9599 rpcz_store.cc:276] 0503 13:58:41.554556 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.022603 (+1468047us) service_pool.cc:224] Handling call
0503 13:58:43.027931 (+ 5328us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:43.032150 (+ 4219us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.032324 (+ 174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:43.031248 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:43.032914 10768 raft_consensus.cc:2804] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:43.033641 10898 consensus_queue.cc:260] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.002944 10636 leader_election.cc:290] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:43.036146 9446 raft_consensus.cc:2468] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
I20260503 13:58:43.038084 10638 raft_consensus.cc:3055] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Stepping down as leader of term 4
I20260503 13:58:43.043130 10638 raft_consensus.cc:740] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 428b342dc4f041df956015e7d7cd173a, State: Running, Role: LEADER
I20260503 13:58:43.015496 9595 rpcz_store.cc:276] 0503 13:58:41.048232 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.682711 (+634479us) service_pool.cc:224] Handling call
0503 13:58:41.744335 (+ 61624us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:42.152137 (+407802us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:42.164185 (+ 12048us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:42.185056 (+ 20871us) log.cc:844] Serialized 52 byte log entry
0503 13:58:42.224206 (+ 39150us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:42.445997 (+221791us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.516329 (+ 70332us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.552763 (+ 36434us) spinlock_profiling.cc:243] Waited 79.9 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:42.563623 (+ 10860us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.563697 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:43.014870 (+451173us) raft_consensus.cc:1685] finished
0503 13:58:43.014901 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:43.015039 (+ 138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":370987,"prepare.queue_time_us":174269,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":72,"raft.queue_time_us":410657,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":15121,"spinlock_wait_cycles":183701888,"thread_start_us":38345,"threads_started":1,"wal-append.queue_time_us":233811}
I20260503 13:58:43.044013 10638 consensus_queue.cc:260] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.009719 10721 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:43.021335 11044 log.cc:927] Time spent T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20260503 13:58:43.046403 10638 raft_consensus.cc:3060] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:43.049422 9444 raft_consensus.cc:1240] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 6. Current term is 7. Ops: []
I20260503 13:58:43.037993 10856 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:43.040642 10768 raft_consensus.cc:493] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:43.057081 9597 raft_consensus.cc:1275] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260503 13:58:43.057703 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 534) took 1504 ms. Trace:
I20260503 13:58:43.058012 9597 rpcz_store.cc:276] 0503 13:58:41.553569 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.055014 (+1501445us) service_pool.cc:224] Handling call
0503 13:58:43.055294 (+ 280us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:43.057401 (+ 2107us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.057648 (+ 247us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:43.045149 10869 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:43.046752 9595 raft_consensus.cc:1275] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260503 13:58:43.059587 10898 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:43.056857 10768 raft_consensus.cc:3060] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:43.054456 10925 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:43.075371 10960 raft_consensus.cc:3060] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:43.065351 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 502) took 2021 ms. Trace:
I20260503 13:58:43.066622 9449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 598) took 1187 ms. Trace:
I20260503 13:58:43.067674 9444 raft_consensus.cc:1275] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260503 13:58:43.077991 9449 rpcz_store.cc:276] 0503 13:58:41.879615 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.989389 (+109774us) service_pool.cc:224] Handling call
0503 13:58:41.989658 (+ 269us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:42.069628 (+ 79970us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.066421 (+996793us) spinlock_profiling.cc:243] Waited 64.4 ms on lock 0x7b54002e63d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:43.066567 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":148187392}
I20260503 13:58:43.042450 10906 raft_consensus.cc:515] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.076786 9594 rpcz_store.cc:276] 0503 13:58:41.044168 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.597559 (+553391us) service_pool.cc:224] Handling call
0503 13:58:41.615209 (+ 17650us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:41.781814 (+166605us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:41.782065 (+ 251us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.231553 (+449488us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.403462 (+171909us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:42.403514 (+ 52us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:42.403551 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.065155 (+661604us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b54002ba7d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:43.065195 (+ 40us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:43.065302 (+ 107us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":374430,"prepare.queue_time_us":39729,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":68,"raft.queue_time_us":425217,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":786,"spinlock_wait_cycles":386285312,"thread_start_us":241535,"threads_started":2,"wal-append.queue_time_us":284768}
I20260503 13:58:43.093950 9597 raft_consensus.cc:1275] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260503 13:58:43.094338 9597 pending_rounds.cc:85] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868: Aborting all ops after (but not including) 4
I20260503 13:58:43.094549 9597 pending_rounds.cc:107] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260503 13:58:43.094761 9597 raft_consensus.cc:2889] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260503 13:58:43.099687 9594 raft_consensus.cc:1275] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:43.100281 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 540) took 1479 ms. Trace:
I20260503 13:58:43.100881 9594 rpcz_store.cc:276] 0503 13:58:41.620953 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.082749 (+1461796us) service_pool.cc:224] Handling call
0503 13:58:43.092969 (+ 10220us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:43.100051 (+ 7082us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.100233 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:43.098912 9405 catalog_manager.cc:6134] T 00000000000000000000000000000000 P 23e911af899f462ba467ed1dfe5a5f91: Tablet 030fcc1b19c3489ea0ca320576ed54ee (table test-workload [id=54209d96daad4531bdc33057c81f7430]) was not created within the allowed timeout. Replacing with a new tablet 0b8deb9fc7614fa9a99164f2adc5be86
W20260503 13:58:43.109153 9494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:43.117883 10871 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:43.118355 10871 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.120265 10871 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:43.082243 10660 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:43.147886 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 539) took 1586 ms. Trace:
I20260503 13:58:43.148334 9597 rpcz_store.cc:276] 0503 13:58:41.561305 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.065585 (+1504280us) service_pool.cc:224] Handling call
0503 13:58:43.079555 (+ 13970us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:43.102965 (+ 23410us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.147818 (+ 44853us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:43.089802 9595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 538) took 1527 ms. Trace:
W20260503 13:58:43.166841 10770 consensus_meta.cc:287] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.607s user 0.007s sys 0.000s
I20260503 13:58:43.166570 9643 consensus_queue.cc:260] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.168272 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 607) took 1116 ms. Trace:
I20260503 13:58:43.172227 10732 raft_consensus.cc:493] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:43.173441 10732 raft_consensus.cc:515] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.170287 10993 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:43.171785 9594 raft_consensus.cc:1275] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260503 13:58:43.169680 10906 leader_election.cc:290] T a5439e0e393b4112a26459eb56502fae P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:43.172859 9448 rpcz_store.cc:276] 0503 13:58:42.051980 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.594351 (+542371us) service_pool.cc:224] Handling call
0503 13:58:42.637000 (+ 42649us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:42.716843 (+ 79843us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:42.716869 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:42.716906 (+ 37us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:42.716933 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.168050 (+451117us) spinlock_profiling.cc:243] Waited 79.7 ms on lock 0x7b540039d5d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:43.168096 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:43.168218 (+ 122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":183229824}
I20260503 13:58:43.175272 9643 ts_tablet_manager.cc:1434] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868: Time spent starting tablet: real 1.195s user 0.016s sys 0.007s
W20260503 13:58:43.183804 9643 ts_tablet_manager.cc:1467] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868: Tablet startup took 1203ms
W20260503 13:58:43.182415 11015 log.cc:927] Time spent T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 1.097s user 0.008s sys 0.005s
I20260503 13:58:43.156344 9595 rpcz_store.cc:276] 0503 13:58:41.560752 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.044721 (+1483969us) service_pool.cc:224] Handling call
0503 13:58:43.046444 (+ 1723us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:43.071894 (+ 25450us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.088389 (+ 16495us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:43.184000 10927 raft_consensus.cc:493] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:43.184515 10927 raft_consensus.cc:515] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.185341 9595 raft_consensus.cc:1275] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260503 13:58:43.186447 10927 leader_election.cc:290] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:43.186635 9595 raft_consensus.cc:1275] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260503 13:58:43.187274 10459 leader_election.cc:290] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:43.187877 9595 raft_consensus.cc:1275] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260503 13:58:43.189656 9643 ts_tablet_manager.cc:1469] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868: Trace:
0503 13:58:05.692411 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:06.287455 (+595044us) service_pool.cc:224] Handling call
0503 13:58:06.306315 (+ 18860us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0503 13:58:06.306398 (+ 83us) ts_tablet_manager.cc:682] Creating new metadata...
0503 13:58:06.415938 (+109540us) tablet_metadata.cc:665] Metadata flushed
0503 13:58:06.452161 (+ 36223us) tablet_replica.cc:206] Creating consensus instance
0503 13:58:06.452533 (+ 372us) inbound_call.cc:177] Queueing success response
0503 13:58:41.225495 (+34772962us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0503 13:58:41.980485 (+754990us) ts_tablet_manager.cc:1435] Starting tablet replica
0503 13:58:42.075303 (+ 94818us) tablet_replica.cc:261] Starting instrumentation
0503 13:58:42.075768 (+ 465us) tablet_replica.cc:280] Starting consensus
0503 13:58:43.166314 (+1090546us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b4c00348000. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3fdfe270 00007f9f3fe0f2f2 00007f9f3fdd5335 00007f9f3fdfc77d 00007f9f3fdfc3e3 00007f9f41b2656b 00007f9f41b27095 00007f9f41b22310 00007f9f41b21020 00007f9f45528f9e 00007f9f46b4e44a
Metrics: {"mutex_wait_us":127507,"raft.queue_time_us":276,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"spinlock_wait_cycles":3409664,"tablet-open.queue_time_us":34773029}
I20260503 13:58:43.191291 10756 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Advancing to term 7
I20260503 13:58:43.191684 9594 raft_consensus.cc:1275] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260503 13:58:43.192566 9450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 592) took 1510 ms. Trace:
I20260503 13:58:43.193735 9450 rpcz_store.cc:276] 0503 13:58:41.682345 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.918043 (+235698us) service_pool.cc:224] Handling call
0503 13:58:41.929887 (+ 11844us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:41.949539 (+ 19652us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:41.949565 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:41.949987 (+ 422us) log.cc:844] Serialized 26 byte log entry
0503 13:58:42.058933 (+108946us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:42.058977 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.059040 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:43.181792 (+1122752us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b24000ef348. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e8dfe94 00007f9f3e8ddf9b 00007f9f3e96cd39 00007f9f3e96c9c5 00007f9f3e96c00d 00007f9f416ae4ae 00007f9f416c5c2c 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1
0503 13:58:43.192286 (+ 10494us) raft_consensus.cc:1685] finished
0503 13:58:43.192307 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:43.192467 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19333,"raft.queue_time_us":22902,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":2621,"spinlock_wait_cycles":5427456,"thread_start_us":108623,"threads_started":1,"wal-append.queue_time_us":120782}
I20260503 13:58:43.204839 9596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 519) took 2155 ms. Trace:
I20260503 13:58:43.205749 9594 raft_consensus.cc:1275] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260503 13:58:43.206730 9595 raft_consensus.cc:1240] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Rejecting Update request from peer 428b342dc4f041df956015e7d7cd173a for earlier term 4. Current term is 5. Ops: [3.2-4.3]
W20260503 13:58:43.207329 11021 log.cc:927] Time spent T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.718s user 0.001s sys 0.007s
I20260503 13:58:43.208240 9595 raft_consensus.cc:1240] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260503 13:58:43.208810 9445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 595) took 1433 ms. Trace:
I20260503 13:58:43.210188 9595 raft_consensus.cc:3060] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868 [term 0 FOLLOWER]: Advancing to term 1
I20260503 13:58:43.209180 9445 rpcz_store.cc:276] 0503 13:58:41.775852 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.965136 (+189284us) service_pool.cc:224] Handling call
0503 13:58:41.969800 (+ 4664us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:41.970166 (+ 366us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:41.970196 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:41.975146 (+ 4950us) log.cc:844] Serialized 52 byte log entry
0503 13:58:42.139951 (+164805us) raft_consensus.cc:1645] Marking committed up to 4
0503 13:58:42.187803 (+ 47852us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.187997 (+ 194us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.188066 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:43.205263 (+1017197us) spinlock_profiling.cc:243] Waited 46 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:43.208489 (+ 3226us) raft_consensus.cc:1685] finished
0503 13:58:43.208621 (+ 132us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:43.208755 (+ 134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":42492,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":2009,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":5305,"spinlock_wait_cycles":105878400,"thread_start_us":71857,"threads_started":1,"wal-append.queue_time_us":403117}
I20260503 13:58:43.210412 10732 leader_election.cc:290] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:43.205210 9596 rpcz_store.cc:276] 0503 13:58:41.049553 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.762756 (+713203us) service_pool.cc:224] Handling call
0503 13:58:41.763027 (+ 271us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:42.202958 (+439931us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:42.202985 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:42.203372 (+ 387us) log.cc:844] Serialized 26 byte log entry
0503 13:58:42.584907 (+381535us) raft_consensus.cc:1645] Marking committed up to 4
0503 13:58:42.654899 (+ 69992us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.655075 (+ 176us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:42.655160 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:43.204647 (+549487us) raft_consensus.cc:1685] finished
0503 13:58:43.204667 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:43.204786 (+ 119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":607013,"prepare.queue_time_us":83257,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":1538,"raft.queue_time_us":481520,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":48320,"spinlock_wait_cycles":6049536,"thread_start_us":381232,"threads_started":1,"wal-append.queue_time_us":977156}
I20260503 13:58:43.211982 9596 raft_consensus.cc:1275] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260503 13:58:43.213066 9596 raft_consensus.cc:1275] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260503 13:58:43.214612 9593 raft_consensus.cc:2468] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:43.214767 9596 raft_consensus.cc:1240] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Rejecting Update request from peer 1cb63886040047559d3893923cc9aaf6 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:43.215947 9596 raft_consensus.cc:1275] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260503 13:58:43.217180 11039 raft_consensus.cc:3055] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Stepping down as leader of term 4
W20260503 13:58:43.221886 9593 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 493) took 1846 ms (client timeout 1573 ms). Trace:
W20260503 13:58:43.222272 9593 rpcz_store.cc:269] 0503 13:58:41.375295 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.391868 (+ 16573us) service_pool.cc:224] Handling call
0503 13:58:43.221709 (+1829841us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b5400038254. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:43.221822 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4159826688}
I20260503 13:58:43.217546 11039 raft_consensus.cc:740] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:43.226400 9594 raft_consensus.cc:1275] T a74decf860264836999f054535dd35ab P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:43.230150 9600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 491) took 2190 ms. Trace:
I20260503 13:58:43.195065 10768 raft_consensus.cc:515] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.270382 11039 consensus_queue.cc:260] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.271723 11039 raft_consensus.cc:3060] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:43.272089 9598 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 7.
I20260503 13:58:43.230508 9600 rpcz_store.cc:276] 0503 13:58:41.039840 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.039949 (+ 109us) service_pool.cc:224] Handling call
0503 13:58:41.052859 (+ 12910us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:41.243472 (+190613us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:41.243498 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:41.243872 (+ 374us) log.cc:844] Serialized 26 byte log entry
0503 13:58:41.395871 (+151999us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:41.415602 (+ 19731us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.492027 (+1076425us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.218542 (+726515us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54002bb6d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:43.218588 (+ 46us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:43.218633 (+ 45us) raft_consensus.cc:1685] finished
0503 13:58:43.229955 (+ 11322us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:43.230097 (+ 142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":880340,"prepare.queue_time_us":19506,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":2112,"raft.queue_time_us":988512,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":17985,"spinlock_wait_cycles":437644160,"thread_start_us":362372,"threads_started":2,"wal-append.queue_time_us":225760,"wal-append.run_cpu_time_us":1860,"wal-append.run_wall_time_us":1076744}
W20260503 13:58:43.279910 9528 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.017s sys 0.016s
W20260503 13:58:43.280309 9528 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.525s user 0.017s sys 0.017s
I20260503 13:58:43.308823 9452 raft_consensus.cc:1275] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260503 13:58:43.236653 10802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.017s sys 0.032s
@ 0x7f9f3ec80cf0 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
W20260503 13:58:43.329608 11023 log.cc:927] Time spent T f5b937be606a421bab59946d15a81744 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.815s user 0.009s sys 0.000s
W20260503 13:58:43.353668 9494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:43.311088 10802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.643s user 0.017s sys 0.032s
I20260503 13:58:43.361050 9595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 502) took 1128 ms. Trace:
I20260503 13:58:43.361445 9595 rpcz_store.cc:276] 0503 13:58:42.232627 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.209839 (+977212us) service_pool.cc:224] Handling call
0503 13:58:43.210083 (+ 244us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:43.360785 (+150702us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.360989 (+ 204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:43.373205 9452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 601) took 1399 ms. Trace:
I20260503 13:58:43.373585 9452 rpcz_store.cc:276] 0503 13:58:41.973522 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.069135 (+ 95613us) service_pool.cc:224] Handling call
0503 13:58:42.069417 (+ 282us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:43.309274 (+1239857us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.372960 (+ 63686us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54004213d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:43.373143 (+ 183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2783826816}
W20260503 13:58:43.375980 9601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.691s user 0.017s sys 0.045s
W20260503 13:58:43.376308 9601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.691s user 0.017s sys 0.045s
W20260503 13:58:43.377934 10733 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.030s sys 0.041s
W20260503 13:58:43.379706 10733 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.662s user 0.030s sys 0.041s
I20260503 13:58:43.380391 9601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 516) took 2331 ms. Trace:
I20260503 13:58:43.380754 9601 rpcz_store.cc:276] 0503 13:58:41.048894 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:41.738281 (+689387us) service_pool.cc:224] Handling call
0503 13:58:41.738594 (+ 313us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:42.086623 (+348029us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:42.086651 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:42.087162 (+ 511us) log.cc:844] Serialized 52 byte log entry
0503 13:58:42.165946 (+ 78784us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:42.566246 (+400300us) spinlock_profiling.cc:243] Waited 98.6 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:42.584409 (+ 18163us) log.cc:844] Serialized 14 byte log entry
0503 13:58:42.673975 (+ 89566us) log.cc:844] Serialized 14 byte log entry
0503 13:58:43.379985 (+706010us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.380072 (+ 87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:43.380199 (+ 127us) raft_consensus.cc:1685] finished
0503 13:58:43.380215 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:43.380336 (+ 121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":562962,"prepare.queue_time_us":124806,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":3285,"raft.queue_time_us":386431,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":7482,"spinlock_wait_cycles":226674432,"thread_start_us":1191269,"threads_started":3,"wal-append.queue_time_us":131884}
W20260503 13:58:43.382443 9526 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.017s sys 0.017s
W20260503 13:58:43.382721 9526 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s user 0.017s sys 0.017s
W20260503 13:58:43.383329 9568 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.017s sys 0.045s
W20260503 13:58:43.383594 9568 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.698s user 0.017s sys 0.045s
W20260503 13:58:43.384631 9567 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.699s user 0.008s sys 0.065s
W20260503 13:58:43.384927 9567 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.700s user 0.008s sys 0.065s
W20260503 13:58:43.385394 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 705278us
W20260503 13:58:43.386068 9524 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.002s sys 0.020s
W20260503 13:58:43.386344 9524 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.557s user 0.002s sys 0.020s
W20260503 13:58:43.391193 9527 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.025s sys 0.017s
W20260503 13:58:43.391462 9527 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.637s user 0.025s sys 0.017s
I20260503 13:58:43.406620 9567 leader_election.cc:304] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:43.407832 9567 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:43.386025 9419 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.755s user 0.010s sys 0.024s
W20260503 13:58:43.410789 9419 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.780s user 0.010s sys 0.024s
W20260503 13:58:43.498787 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 73437.5us
I20260503 13:58:43.500960 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:43.510314 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:43.508180 10871 consensus_queue.cc:1048] T b1dfd5295058420a9da1c1080e5caef3 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260503 13:58:43.509076 10638 consensus_queue.cc:1048] T 306069d785d24834b84d590925e644cf P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:58:43.513495 9598 raft_consensus.cc:2393] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 5: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
W20260503 13:58:43.462378 11043 log.cc:927] Time spent T 83ebc0b241994520929db0a8b7fb8546 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20260503 13:58:43.515187 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54bf4ee675fb4fcf800b317817906dcc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:43.518100 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:43.520753 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:43.520983 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 113231us
I20260503 13:58:43.522989 9567 leader_election.cc:304] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:43.522243 9600 raft_consensus.cc:2468] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:43.524873 9601 raft_consensus.cc:1275] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260503 13:58:43.526463 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:43.527382 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:43.527297 9567 leader_election.cc:304] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:43.528891 9598 raft_consensus.cc:2468] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 2.
I20260503 13:58:43.511124 10884 raft_consensus.cc:493] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:43.511124 10445 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:43.422403 10770 consensus_queue.cc:1048] T 22875ad0517447a2a9afc25bf367486f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.113s
I20260503 13:58:43.552057 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:43.463333 11053 raft_consensus.cc:2749] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:43.462785 10993 consensus_queue.cc:1048] T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.108s
I20260503 13:58:43.421921 10650 consensus_queue.cc:1048] T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.106s
I20260503 13:58:43.513226 10867 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:43.534322 10880 consensus_queue.cc:1048] T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260503 13:58:43.555785 10852 consensus_queue.cc:1048] T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.002s
I20260503 13:58:43.520274 10899 consensus_queue.cc:1048] T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.045s
I20260503 13:58:43.519965 10851 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:43.534574 9494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:43.564646 10654 raft_consensus.cc:493] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:43.514262 10544 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:43.520540 11040 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:43.552066 10733 raft_consensus.cc:2804] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election won for term 6
I20260503 13:58:43.550851 10756 raft_consensus.cc:3060] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:43.519912 10907 consensus_queue.cc:1048] T b3b7ab9cca2847d392329f5c80956bbd P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.046s
I20260503 13:58:43.529145 11050 raft_consensus.cc:2749] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260503 13:58:43.560663 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:43.526173 10847 consensus_queue.cc:1048] T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.041s
I20260503 13:58:43.554107 10856 consensus_queue.cc:1048] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
W20260503 13:58:43.621038 9492 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.006s sys 0.030s
W20260503 13:58:43.634670 9492 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.581s user 0.006s sys 0.031s
I20260503 13:58:43.597605 10768 leader_election.cc:290] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:43.556969 10906 consensus_queue.cc:1048] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260503 13:58:43.562451 10884 raft_consensus.cc:515] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.598737 10868 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:43.634976 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 589454us
I20260503 13:58:43.556851 10946 consensus_queue.cc:1048] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260503 13:58:43.562354 10445 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.621001 11051 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:43.656601 9492 leader_election.cc:304] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:43.593776 10867 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.628340 11053 raft_consensus.cc:493] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:43.640241 10981 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:43.640125 10898 raft_consensus.cc:493] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:43.623365 10638 consensus_queue.cc:1048] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.041s
I20260503 13:58:43.641047 10733 raft_consensus.cc:697] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [term 6 LEADER]: Becoming Leader. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:43.682065 10193 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:43.655450 10295 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:43.683120 11050 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:43.621642 10851 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.624624 10654 raft_consensus.cc:515] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.628155 10871 consensus_queue.cc:1048] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.042s
I20260503 13:58:43.682618 10920 raft_consensus.cc:493] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:43.668744 10993 raft_consensus.cc:493] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:43.701076 9599 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.002s sys 0.014s
I20260503 13:58:43.632016 11040 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:43.701370 9599 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.651s user 0.002s sys 0.014s
I20260503 13:58:43.632416 10544 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.680915 10880 raft_consensus.cc:493] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:43.700371 9602 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.003s sys 0.016s
W20260503 13:58:43.710109 9602 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s user 0.004s sys 0.016s
W20260503 13:58:43.701733 9566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.038s user 0.010s sys 0.014s
W20260503 13:58:43.717434 9566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.054s user 0.010s sys 0.015s
W20260503 13:58:43.717727 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 1.05982e+06us
W20260503 13:58:43.702018 9451 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.070s user 0.019s sys 0.056s
I20260503 13:58:43.719118 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
W20260503 13:58:43.724593 9451 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.092s user 0.019s sys 0.056s
W20260503 13:58:43.702303 9453 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.006s sys 0.010s
W20260503 13:58:43.728039 9453 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.657s user 0.006s sys 0.010s
I20260503 13:58:43.687126 10898 raft_consensus.cc:515] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.738608 9602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 492) took 2363 ms. Trace:
I20260503 13:58:43.744283 9448 raft_consensus.cc:1275] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:43.745249 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:43.691548 10193 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.710457 10891 raft_consensus.cc:493] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:43.700693 9418 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.990s user 0.000s sys 0.016s
W20260503 13:58:43.700631 11042 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.012s sys 0.003s
W20260503 13:58:43.708355 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260503 13:58:43.689414 10852 raft_consensus.cc:2804] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260503 13:58:43.716593 9416 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.031s user 0.011s sys 0.004s
I20260503 13:58:43.661386 10868 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.726794 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:43.668426 11051 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.703892 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 536) took 2145 ms. Trace:
I20260503 13:58:43.745266 9445 raft_consensus.cc:1240] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5. Ops: []
W20260503 13:58:43.750682 11042 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.746s user 0.013s sys 0.003s
W20260503 13:58:43.748449 9418 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.044s user 0.000s sys 0.017s
I20260503 13:58:43.751358 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:43.693928 10295 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.710912 10993 raft_consensus.cc:515] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:43.766345 9523 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.056s user 0.031s sys 0.107s
I20260503 13:58:43.685472 10981 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.709510 10768 consensus_queue.cc:1048] T 11b8ecb173e84681b8fb389c89395838 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260503 13:58:43.760914 9599 rpcz_store.cc:276] 0503 13:58:41.558076 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.035682 (+1477606us) service_pool.cc:224] Handling call
0503 13:58:43.047336 (+ 11654us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:43.047534 (+ 198us) raft_consensus.cc:1534] Early marking committed up to index 1
0503 13:58:43.050607 (+ 3073us) log.cc:844] Serialized 14 byte log entry
0503 13:58:43.703568 (+652961us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:43.703622 (+ 54us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:43.703661 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.703724 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:43.703836 (+ 112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27026,"prepare.queue_time_us":5535,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":1794,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":5708,"thread_start_us":650815,"threads_started":1}
W20260503 13:58:43.765683 9418 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:43.797966 10295 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:43.766017 10885 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.006s sys 0.010s
I20260503 13:58:43.785373 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:43.754614 9416 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.069s user 0.011s sys 0.004s
I20260503 13:58:43.710294 10920 raft_consensus.cc:515] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.787680 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:43.788492 9523 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.079s user 0.032s sys 0.107s
W20260503 13:58:43.792479 9419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:43.829794 10639 raft_consensus.cc:2804] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260503 13:58:43.829869 9523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 636) took 1159 ms. Trace:
I20260503 13:58:43.830250 9523 rpcz_store.cc:276] 0503 13:58:42.670292 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.671212 (+ 920us) service_pool.cc:224] Handling call
0503 13:58:42.702149 (+ 30937us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:42.702350 (+ 201us) raft_consensus.cc:1534] Early marking committed up to index 4
0503 13:58:42.709176 (+ 6826us) log.cc:844] Serialized 14 byte log entry
0503 13:58:43.829558 (+1120382us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:43.829606 (+ 48us) raft_consensus.cc:1645] Marking committed up to 4
0503 13:58:43.829647 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.829708 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:43.829816 (+ 108us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":408316,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"thread_start_us":1120076,"threads_started":1}
I20260503 13:58:43.831434 10639 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:43.831840 10639 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Advancing to term 8
I20260503 13:58:43.833532 10851 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:43.834115 10867 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:43.756932 10998 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.255s user 0.019s sys 0.021s
I20260503 13:58:43.755049 10891 raft_consensus.cc:515] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.754738 10733 consensus_queue.cc:237] T 973019c2128c4039aa1d442714f23ae7 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.829912 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:43.739029 9602 rpcz_store.cc:276] 0503 13:58:41.375081 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.756273 (+1381192us) service_pool.cc:224] Handling call
0503 13:58:42.778075 (+ 21802us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:42.778261 (+ 186us) raft_consensus.cc:1534] Early marking committed up to index 1
0503 13:58:42.941713 (+163452us) log.cc:844] Serialized 14 byte log entry
0503 13:58:43.738273 (+796560us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:43.738331 (+ 58us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:43.738374 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.738438 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:43.738551 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":159909,"prepare.queue_time_us":163689,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":9337,"raft.queue_time_us":27549,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":7703,"thread_start_us":768691,"threads_started":1}
I20260503 13:58:43.796736 10981 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:43.696451 11050 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.672807 11053 raft_consensus.cc:515] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:43.807925 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d04573f959de406a9590a1603f633e7e" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:43.808262 10295 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:43.808756 9566 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:43.757200 10879 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.007s sys 0.009s
I20260503 13:58:43.754069 10852 raft_consensus.cc:493] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:43.722882 10880 raft_consensus.cc:515] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:43.822364 9416 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:43.812597 10802 consensus_queue.cc:1048] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260503 13:58:43.825533 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:43.872830 10850 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.751s user 0.005s sys 0.019s
I20260503 13:58:43.819577 10445 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:43.928061 9527 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.017s sys 0.047s
W20260503 13:58:43.869434 9596 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.007s sys 0.010s
W20260503 13:58:44.007375 9596 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s user 0.009s sys 0.010s
W20260503 13:58:43.766729 11057 log.cc:927] Time spent T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.215s user 0.000s sys 0.001s
W20260503 13:58:43.871193 10849 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.837s user 0.000s sys 0.021s
W20260503 13:58:43.835784 11039 consensus_meta.cc:287] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.564s user 0.006s sys 0.003s
W20260503 13:58:43.820737 10885 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s user 0.006s sys 0.010s
W20260503 13:58:43.839223 11067 log.cc:927] Time spent T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
W20260503 13:58:43.896428 10998 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.395s user 0.019s sys 0.028s
I20260503 13:58:43.991390 10193 leader_election.cc:290] T a17efa5a2f524bcb966ff62c4182503f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:43.913367 10898 leader_election.cc:290] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:44.023713 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 588) took 1318 ms. Trace:
I20260503 13:58:44.024047 9526 rpcz_store.cc:276] 0503 13:58:42.704818 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.705312 (+ 494us) service_pool.cc:224] Handling call
0503 13:58:42.706906 (+ 1594us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:42.708899 (+ 1993us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:42.708927 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:42.709297 (+ 370us) log.cc:844] Serialized 26 byte log entry
0503 13:58:43.383039 (+673742us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:43.383076 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.383136 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:44.023507 (+640371us) raft_consensus.cc:1685] finished
0503 13:58:44.023528 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:44.023659 (+ 131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1648,"raft.queue_time_us":3567,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":4526,"thread_start_us":562193,"threads_started":1,"wal-append.queue_time_us":624684}
I20260503 13:58:43.905534 9416 leader_election.cc:304] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
W20260503 13:58:43.896420 9597 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.721s user 0.006s sys 0.020s
W20260503 13:58:44.028847 9597 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.853s user 0.006s sys 0.020s
I20260503 13:58:43.828408 10868 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:43.819490 10884 leader_election.cc:290] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:43.993023 11072 log.cc:927] Time spent T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260503 13:58:43.904946 10295 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.007725 9566 leader_election.cc:304] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:44.007180 9527 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s user 0.018s sys 0.060s
I20260503 13:58:43.809525 10732 raft_consensus.cc:493] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:43.920148 9452 raft_consensus.cc:1275] T 50d2bf9820c443d5b7d017429cb0a818 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260503 13:58:43.896750 10649 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.013s sys 0.012s
W20260503 13:58:43.881845 9446 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.778s user 0.003s sys 0.019s
I20260503 13:58:43.904807 10852 raft_consensus.cc:3060] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Advancing to term 4
W20260503 13:58:43.886202 10916 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.440s user 0.008s sys 0.024s
W20260503 13:58:43.892055 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 82864.4us
I20260503 13:58:43.977747 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:43.906863 10981 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:44.009850 9447 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.980s user 0.004s sys 0.019s
W20260503 13:58:44.035689 9447 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.006s user 0.004s sys 0.019s
W20260503 13:58:44.010110 9493 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.245s user 0.015s sys 0.010s
W20260503 13:58:44.037705 9493 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.273s user 0.015s sys 0.010s
I20260503 13:58:44.035486 9444 raft_consensus.cc:2468] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:43.896303 11040 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:44.018374 9449 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.938s user 0.005s sys 0.022s
I20260503 13:58:43.898895 10654 leader_election.cc:290] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:43.902719 10879 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s user 0.007s sys 0.011s
I20260503 13:58:44.021924 10802 raft_consensus.cc:493] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:44.007145 11056 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:44.029160 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 123710us
I20260503 13:58:44.029457 10885 raft_consensus.cc:2749] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260503 13:58:43.913450 10639 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:44.012640 10850 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.885s user 0.016s sys 0.032s
I20260503 13:58:44.029672 11050 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:43.904253 10946 raft_consensus.cc:493] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:44.006328 11037 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:43.913785 10925 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:44.032943 9446 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.930s user 0.003s sys 0.020s
I20260503 13:58:44.034318 9527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 637) took 1356 ms. Trace:
I20260503 13:58:43.992305 11051 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:43.994174 11053 leader_election.cc:290] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:44.016497 10849 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.982s user 0.000s sys 0.023s
I20260503 13:58:44.036075 9566 leader_election.cc:304] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:44.040014 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:44.037506 10898 raft_consensus.cc:2749] T 3b48fff133f84ff7827806460a23803b P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260503 13:58:44.060451 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:44.031212 10732 raft_consensus.cc:515] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.042603 9416 leader_election.cc:304] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:44.063387 9416 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:44.064576 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3d3814bacf748a4928419aa5e860f4a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.064841 9416 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:44.065168 9524 raft_consensus.cc:2468] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:44.066290 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.040830 10193 consensus_queue.cc:1048] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.026s
I20260503 13:58:44.066815 9524 raft_consensus.cc:2468] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:44.048353 11056 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:44.032828 10649 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s user 0.013s sys 0.013s
I20260503 13:58:44.039798 10638 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:44.043401 9418 leader_election.cc:304] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:44.034883 10916 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.591s user 0.008s sys 0.024s
I20260503 13:58:44.032327 10993 leader_election.cc:290] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:44.069900 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.061813 9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3500374093e84b968ed96ed4dbdfda69" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.062247 9566 leader_election.cc:304] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
W20260503 13:58:44.041046 9449 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.961s user 0.005s sys 0.022s
I20260503 13:58:44.057380 11037 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:44.068001 9491 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.005s sys 0.021s
I20260503 13:58:44.068832 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010e981b4bda450e9e11c3f6ff93f7eb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.056934 9527 rpcz_store.cc:276] 0503 13:58:42.677909 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.678142 (+ 233us) service_pool.cc:224] Handling call
0503 13:58:42.703928 (+ 25786us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:42.704097 (+ 169us) raft_consensus.cc:1534] Early marking committed up to index 4
0503 13:58:43.391729 (+687632us) log.cc:844] Serialized 14 byte log entry
0503 13:58:44.031688 (+639959us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:44.031742 (+ 54us) raft_consensus.cc:1645] Marking committed up to 4
0503 13:58:44.031781 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:44.031857 (+ 76us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:44.034263 (+ 2406us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10280,"prepare.queue_time_us":406800,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":484,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":9693,"thread_start_us":1276764,"threads_started":2}
I20260503 13:58:44.036998 10544 leader_election.cc:290] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:44.129559 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 48454.3us
I20260503 13:58:44.131222 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d04573f959de406a9590a1603f633e7e" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.131640 10639 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 8 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:44.131965 9519 raft_consensus.cc:2393] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 1: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:44.132138 11053 raft_consensus.cc:2749] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260503 13:58:44.123816 10998 raft_consensus.cc:2804] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:44.132632 11059 consensus_queue.cc:1048] T 3046bdc111c5477eb4eabf09e7f4e755 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260503 13:58:44.133076 10998 raft_consensus.cc:493] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:44.133450 10998 raft_consensus.cc:3060] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:44.046490 10802 raft_consensus.cc:515] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.059628 10920 leader_election.cc:290] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:44.122335 9491 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.569s user 0.013s sys 0.025s
I20260503 13:58:44.143513 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26449f3ab09141ada50d9334532cb3ec" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:44.154318 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 601136us
W20260503 13:58:44.156303 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 81704.8us
I20260503 13:58:44.157196 9566 leader_election.cc:304] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:44.067770 10885 raft_consensus.cc:2749] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260503 13:58:44.068495 11039 raft_consensus.cc:2804] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:44.067528 10650 raft_consensus.cc:493] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:44.061004 11069 raft_consensus.cc:493] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:44.061087 10879 raft_consensus.cc:2749] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260503 13:58:44.058771 10946 raft_consensus.cc:515] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.163093 9491 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:44.068516 10884 raft_consensus.cc:2749] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260503 13:58:44.158380 9416 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:44.069828 10445 raft_consensus.cc:493] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:44.075781 10896 raft_consensus.cc:493] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:44.167405 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.061931 10925 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:44.161675 10636 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.117s user 0.008s sys 0.039s
W20260503 13:58:44.201838 10636 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.157s user 0.008s sys 0.039s
I20260503 13:58:44.173048 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.159660 11059 raft_consensus.cc:493] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:44.156608 10867 consensus_queue.cc:1048] T 754ac001a92e4dd6a0b6620803b83722 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20260503 13:58:44.167215 10654 raft_consensus.cc:2804] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260503 13:58:44.211071 9528 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.930s user 0.044s sys 0.065s
I20260503 13:58:44.212054 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:44.212914 9528 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.931s user 0.044s sys 0.065s
I20260503 13:58:44.168969 10920 consensus_queue.cc:1048] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260503 13:58:44.210302 10654 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:44.215299 10654 raft_consensus.cc:3060] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:44.198835 9491 leader_election.cc:304] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a; no voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:44.200682 10998 raft_consensus.cc:515] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.074362 10638 raft_consensus.cc:515] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.173336 9527 raft_consensus.cc:1240] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: []
I20260503 13:58:44.163656 11039 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:44.131858 10732 leader_election.cc:290] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:44.177946 10756 consensus_meta.cc:287] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.544s user 0.008s sys 0.000s
I20260503 13:58:44.169572 10544 raft_consensus.cc:493] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:44.173452 10891 leader_election.cc:290] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:44.043732 10981 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.183853 10802 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:44.184311 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.185712 10993 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260503 13:58:44.217278 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.208815 10851 raft_consensus.cc:2804] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader election won for term 5
I20260503 13:58:44.164824 10869 consensus_queue.cc:1048] T a74decf860264836999f054535dd35ab P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.046s
I20260503 13:58:44.168359 10960 consensus_queue.cc:1048] T 010e981b4bda450e9e11c3f6ff93f7eb P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260503 13:58:44.222638 10868 raft_consensus.cc:3060] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:44.223156 9451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 605) took 2194 ms. Trace:
I20260503 13:58:44.257570 9451 rpcz_store.cc:276] 0503 13:58:42.028235 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.565222 (+536987us) service_pool.cc:224] Handling call
0503 13:58:42.565474 (+ 252us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:42.565659 (+ 185us) raft_consensus.cc:1534] Early marking committed up to index 1
0503 13:58:42.565681 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:42.571240 (+ 5559us) log.cc:844] Serialized 26 byte log entry
0503 13:58:43.726221 (+1154981us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:43.726259 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.726323 (+ 64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:44.222932 (+496609us) raft_consensus.cc:1685] finished
0503 13:58:44.222954 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:44.223079 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1093971,"threads_started":1,"wal-append.queue_time_us":1632680}
I20260503 13:58:44.171787 10650 raft_consensus.cc:515] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.227140 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.226084 10295 leader_election.cc:290] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:44.229909 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.211022 10880 leader_election.cc:290] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:44.235416 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.174558 11069 raft_consensus.cc:515] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.205880 10445 raft_consensus.cc:515] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.242749 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.260176 10851 raft_consensus.cc:697] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [term 5 LEADER]: Becoming Leader. State: Replica: 1cb63886040047559d3893923cc9aaf6, State: Running, Role: LEADER
I20260503 13:58:44.207262 10896 raft_consensus.cc:515] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.274263 10445 leader_election.cc:290] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:44.274706 10896 leader_election.cc:290] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:44.273276 9520 raft_consensus.cc:2468] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 3.
I20260503 13:58:44.253782 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.275918 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.276974 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.277645 9520 raft_consensus.cc:2393] T 50d2bf9820c443d5b7d017429cb0a818 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 6: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:44.278272 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.278812 9520 raft_consensus.cc:2468] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 2.
I20260503 13:58:44.279292 9519 raft_consensus.cc:1275] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260503 13:58:44.273749 10998 leader_election.cc:290] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:44.279501 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.280050 11069 leader_election.cc:290] T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:44.280438 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.281272 9520 raft_consensus.cc:2468] T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:44.236219 9528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 638) took 1547 ms. Trace:
I20260503 13:58:44.228169 10756 raft_consensus.cc:2749] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:44.227360 11039 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:44.214617 11059 raft_consensus.cc:515] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.261930 9522 raft_consensus.cc:2410] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260503 13:58:44.282205 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.266899 9525 raft_consensus.cc:1240] T 05ba056a4d0c406a881cc34cb66078bc P 428b342dc4f041df956015e7d7cd173a [term 8 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 7. Current term is 8. Ops: []
I20260503 13:58:44.269805 9567 leader_election.cc:304] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6
I20260503 13:58:44.281790 9528 rpcz_store.cc:276] 0503 13:58:42.689346 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:42.704336 (+ 14990us) service_pool.cc:224] Handling call
0503 13:58:42.704628 (+ 292us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:42.704804 (+ 176us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:43.280756 (+575952us) log.cc:844] Serialized 14 byte log entry
0503 13:58:44.235892 (+955136us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:44.235949 (+ 57us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:44.235995 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:44.236062 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:44.236166 (+ 104us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67674,"prepare.queue_time_us":406395,"prepare.run_cpu_time_us":21,"prepare.run_wall_time_us":21,"raft.queue_time_us":23349,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":3879,"thread_start_us":1461824,"threads_started":2}
I20260503 13:58:44.255441 9491 leader_election.cc:304] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:44.264961 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.273865 10851 consensus_queue.cc:237] T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.286084 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.286620 9520 raft_consensus.cc:2393] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 6: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:44.283986 9567 leader_election.cc:304] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:44.288178 9567 leader_election.cc:304] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:44.289311 9567 leader_election.cc:304] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:44.254395 10993 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:44.293203 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.291278 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.295102 9520 raft_consensus.cc:2393] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 5: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:44.296139 9520 raft_consensus.cc:1275] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Refusing update from remote peer 1cb63886040047559d3893923cc9aaf6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260503 13:58:44.298024 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:44.300215 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 630) took 1296 ms. Trace:
I20260503 13:58:44.300580 9453 rpcz_store.cc:276] 0503 13:58:43.003480 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.069497 (+ 66017us) service_pool.cc:224] Handling call
0503 13:58:43.069750 (+ 253us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:43.070033 (+ 283us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:43.070081 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:43.070534 (+ 453us) log.cc:844] Serialized 26 byte log entry
0503 13:58:43.728447 (+657913us) raft_consensus.cc:1645] Marking committed up to 4
0503 13:58:43.728488 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:43.728573 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:44.300015 (+571442us) raft_consensus.cc:1685] finished
0503 13:58:44.300036 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:44.300158 (+ 122us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3595,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":69,"thread_start_us":657645,"threads_started":1,"wal-append.queue_time_us":1145899}
I20260503 13:58:44.290889 10871 raft_consensus.cc:2804] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:44.302493 10871 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:44.302806 10871 raft_consensus.cc:3060] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:44.291128 10732 raft_consensus.cc:2749] T 3500374093e84b968ed96ed4dbdfda69 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260503 13:58:44.304665 10920 raft_consensus.cc:2749] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260503 13:58:44.291263 10193 raft_consensus.cc:2749] T d04573f959de406a9590a1603f633e7e P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260503 13:58:44.316969 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.318822 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.235984 10544 raft_consensus.cc:515] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.321597 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.345039 9523 raft_consensus.cc:2468] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:44.291466 10891 raft_consensus.cc:2804] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260503 13:58:44.316294 11042 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.013s sys 0.043s
I20260503 13:58:44.353121 9524 raft_consensus.cc:2468] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
W20260503 13:58:44.353965 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 63382.7us
I20260503 13:58:44.354722 9567 leader_election.cc:304] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:44.355885 9567 leader_election.cc:304] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:44.282701 10907 consensus_queue.cc:1048] T a763e2ddf6e7463bb123ce8d870a1afb P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.065s
W20260503 13:58:44.356630 11042 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s user 0.013s sys 0.048s
I20260503 13:58:44.359588 11059 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:44.359891 10639 raft_consensus.cc:493] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:44.349570 10891 raft_consensus.cc:493] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:44.361472 10638 leader_election.cc:290] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:44.362347 10879 raft_consensus.cc:2749] T 010e981b4bda450e9e11c3f6ff93f7eb P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260503 13:58:44.306797 11056 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:44.294337 10993 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Advancing to term 7
W20260503 13:58:44.374495 9405 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.016s sys 0.072s
W20260503 13:58:44.375572 9405 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.691s user 0.017s sys 0.072s
I20260503 13:58:44.368814 10891 raft_consensus.cc:3060] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:44.386984 9446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 623) took 1803 ms. Trace:
I20260503 13:58:44.388720 9446 rpcz_store.cc:276] 0503 13:58:42.583549 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.052119 (+468570us) service_pool.cc:224] Handling call
0503 13:58:43.072341 (+ 20222us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:43.078189 (+ 5848us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:43.078216 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:43.078588 (+ 372us) log.cc:844] Serialized 26 byte log entry
0503 13:58:44.059921 (+981333us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:44.060185 (+ 264us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:44.060259 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:44.386758 (+326499us) raft_consensus.cc:1685] finished
0503 13:58:44.386789 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:44.386925 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5496,"raft.queue_time_us":6476,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":1737,"thread_start_us":954046,"threads_started":1,"wal-append.queue_time_us":1290740}
I20260503 13:58:44.367691 11037 leader_election.cc:290] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:44.369277 10946 leader_election.cc:290] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:44.386934 10879 raft_consensus.cc:2749] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:44.398121 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.396507 9567 leader_election.cc:304] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:44.368774 10639 raft_consensus.cc:515] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:44.399595 9492 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.709s user 0.030s sys 0.026s
W20260503 13:58:44.399926 9492 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.712s user 0.030s sys 0.026s
W20260503 13:58:44.400224 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 712141us
I20260503 13:58:44.396183 10925 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:44.388365 11056 consensus_queue.cc:1059] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260503 13:58:44.402344 10295 raft_consensus.cc:2804] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:44.414757 10981 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:44.520358 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 86442.4us
W20260503 13:58:44.549480 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 70377.6us
I20260503 13:58:44.402489 10946 consensus_queue.cc:1048] T 0f951d26e4ed4c3bb307eb276954663b P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.116s
W20260503 13:58:44.558882 11082 log.cc:927] Time spent T 26449f3ab09141ada50d9334532cb3ec P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260503 13:58:44.477216 10998 raft_consensus.cc:493] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:44.559268 10639 leader_election.cc:290] T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:44.565057 11064 raft_consensus.cc:493] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:44.563588 11087 log.cc:927] Time spent T 83ebc0b241994520929db0a8b7fb8546 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20260503 13:58:44.476403 11037 consensus_queue.cc:1059] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
I20260503 13:58:44.478101 10295 raft_consensus.cc:493] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:44.522156 10650 leader_election.cc:290] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:44.518324 9494 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.824s user 0.024s sys 0.063s
W20260503 13:58:44.567812 9448 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.745s user 0.006s sys 0.017s
W20260503 13:58:44.603132 9450 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.845s user 0.017s sys 0.021s
I20260503 13:58:44.625137 9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.666921 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.625847 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.619098 10802 raft_consensus.cc:493] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:44.667536 9523 raft_consensus.cc:2468] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
W20260503 13:58:44.668012 9450 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.910s user 0.020s sys 0.021s
I20260503 13:58:44.667804 9522 raft_consensus.cc:2410] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260503 13:58:44.557744 10925 raft_consensus.cc:3055] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Stepping down as leader of term 5
I20260503 13:58:44.624590 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.670621 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "754ac001a92e4dd6a0b6620803b83722" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:44.562868 10852 consensus_meta.cc:287] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 0.519s user 0.006s sys 0.000s
I20260503 13:58:44.564833 10544 leader_election.cc:290] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:44.565152 10445 raft_consensus.cc:493] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
W20260503 13:58:44.644465 9493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260503 13:58:44.620537 11059 raft_consensus.cc:493] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:44.663710 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:44.667076 9448 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.844s user 0.006s sys 0.017s
I20260503 13:58:44.622463 10193 raft_consensus.cc:493] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:44.671470 9528 raft_consensus.cc:2410] T 754ac001a92e4dd6a0b6620803b83722 P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260503 13:58:44.624529 10867 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:44.639029 11064 raft_consensus.cc:515] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.753180 10867 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:44.753993 11074 log.cc:927] Time spent T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.275s user 0.001s sys 0.000s
@ 0x7f9f3ec73dcd testing::UnitTest::Run() at ??:0
I20260503 13:58:44.754976 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 615) took 2264 ms. Trace:
I20260503 13:58:44.624776 10885 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:44.755174 10867 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:44.666862 9494 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.978s user 0.024s sys 0.066s
I20260503 13:58:44.670643 10925 raft_consensus.cc:740] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:44.745431 10802 raft_consensus.cc:515] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.756808 10925 consensus_queue.cc:260] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.757908 10925 raft_consensus.cc:3060] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:44.752231 10193 raft_consensus.cc:515] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.762310 10193 leader_election.cc:290] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:44.755940 10867 raft_consensus.cc:493] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:44.763365 10867 raft_consensus.cc:515] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:44.765038 11070 log.cc:927] Time spent T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.207s user 0.000s sys 0.000s
I20260503 13:58:44.766043 9449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 633) took 1734 ms. Trace:
W20260503 13:58:44.766348 11084 log.cc:927] Time spent T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.198s user 0.000s sys 0.000s
W20260503 13:58:44.767230 11083 log.cc:927] Time spent T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.248s user 0.000s sys 0.000s
I20260503 13:58:44.766362 9449 rpcz_store.cc:276] 0503 13:58:43.031150 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.078936 (+ 47786us) service_pool.cc:224] Handling call
0503 13:58:43.079201 (+ 265us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:43.079485 (+ 284us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:43.079509 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:43.079885 (+ 376us) log.cc:844] Serialized 26 byte log entry
0503 13:58:44.072019 (+992134us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:44.072057 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:44.072692 (+ 635us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:44.765819 (+693127us) raft_consensus.cc:1685] finished
0503 13:58:44.765839 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:44.765973 (+ 134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":880,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":3101,"thread_start_us":991869,"threads_started":1,"wal-append.queue_time_us":1320797}
I20260503 13:58:44.767946 10871 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.768771 10867 leader_election.cc:290] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:44.773092 11081 log.cc:927] Time spent T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20260503 13:58:44.778666 11051 raft_consensus.cc:3055] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 7 LEADER]: Stepping down as leader of term 7
I20260503 13:58:44.779042 11051 raft_consensus.cc:740] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
I20260503 13:58:44.779819 11051 consensus_queue.cc:260] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.781198 10885 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.782996 10885 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:44.783635 10654 consensus_meta.cc:287] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.568s user 0.003s sys 0.002s
I20260503 13:58:44.785161 11059 raft_consensus.cc:515] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.787030 11059 leader_election.cc:290] T 22875ad0517447a2a9afc25bf367486f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:44.753144 10544 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:44.784019 10654 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
@ 0x378fbc RUN_ALL_TESTS() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:?
I20260503 13:58:44.664543 10295 raft_consensus.cc:3060] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Advancing to term 3
W20260503 13:58:44.752754 11086 log.cc:927] Time spent T 322e401806f7452c9714ce855e35ca9b P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260503 13:58:44.789450 9447 rpcz_store.cc:276] 0503 13:58:42.490245 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.006560 (+516315us) service_pool.cc:224] Handling call
0503 13:58:43.006835 (+ 275us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:43.028990 (+ 22155us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:43.029043 (+ 53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:43.029455 (+ 412us) log.cc:844] Serialized 26 byte log entry
0503 13:58:44.036068 (+1006613us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:44.036115 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:44.036177 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:44.753505 (+717328us) spinlock_profiling.cc:243] Waited 3.05 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:44.754704 (+ 1199us) raft_consensus.cc:1685] finished
0503 13:58:44.754729 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:44.754921 (+ 192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28546,"raft.queue_time_us":31213,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":5505,"spinlock_wait_cycles":7022976,"thread_start_us":1006018,"threads_started":1,"wal-append.queue_time_us":1368290}
W20260503 13:58:44.792194 9526 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.000s sys 0.025s
W20260503 13:58:44.802479 9526 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s user 0.000s sys 0.025s
I20260503 13:58:44.787556 10993 raft_consensus.cc:515] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.796823 9567 leader_election.cc:304] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:44.797686 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:44.747291 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 78688.6us
I20260503 13:58:44.628491 10998 raft_consensus.cc:515] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.755081 11064 leader_election.cc:290] T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
@ 0x377b8d main at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:?
I20260503 13:58:44.750963 11037 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
W20260503 13:58:44.793917 11039 consensus_meta.cc:287] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6: Time spent flushing consensus metadata: real 0.512s user 0.002s sys 0.006s
I20260503 13:58:44.670228 10639 consensus_queue.cc:1048] T c4a7792770d1446eafce219f5f28d7db P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.132s
W20260503 13:58:44.793668 10849 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s user 0.004s sys 0.021s
I20260503 13:58:44.821286 9523 raft_consensus.cc:1240] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:44.822928 9525 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 7.
W20260503 13:58:44.810462 10733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.009s sys 0.041s
I20260503 13:58:44.792153 10544 raft_consensus.cc:515] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:44.820859 10636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.014s sys 0.014s
I20260503 13:58:44.813110 10896 raft_consensus.cc:2804] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:44.813290 10768 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:44.788655 10445 raft_consensus.cc:515] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.840687 9597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 543) took 2613 ms. Trace:
W20260503 13:58:44.825270 10849 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.753s user 0.004s sys 0.022s
W20260503 13:58:44.829452 9565 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:44.828994 10733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s user 0.009s sys 0.041s
I20260503 13:58:44.748011 10852 raft_consensus.cc:515] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.846081 9597 rpcz_store.cc:276] 0503 13:58:42.226830 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.149087 (+922257us) service_pool.cc:224] Handling call
0503 13:58:43.149432 (+ 345us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:43.149817 (+ 385us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:43.149843 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:43.175140 (+ 25297us) log.cc:844] Serialized 26 byte log entry
0503 13:58:44.029190 (+854050us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:44.029229 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:44.029295 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:44.840476 (+811181us) raft_consensus.cc:1685] finished
0503 13:58:44.840493 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:44.840634 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":89,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":6205,"thread_start_us":853717,"threads_started":1,"wal-append.queue_time_us":1258697}
I20260503 13:58:44.816274 11051 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Advancing to term 8
I20260503 13:58:44.849496 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:44.851899 9399 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260503 13:58:44.844569 9566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.687s user 0.008s sys 0.023s
W20260503 13:58:44.881045 9595 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.009s sys 0.018s
W20260503 13:58:44.976922 9566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.819s user 0.008s sys 0.023s
W20260503 13:58:44.899487 9593 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.012s sys 0.017s
W20260503 13:58:44.884987 9602 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.027s sys 0.022s
I20260503 13:58:44.900529 10802 leader_election.cc:290] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:44.977006 9595 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.679s user 0.009s sys 0.027s
I20260503 13:58:44.850060 10650 consensus_queue.cc:1048] T b07d47a095894b958145d029c84c9123 P 428b342dc4f041df956015e7d7cd173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260503 13:58:44.854797 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030fcc1b19c3489ea0ca320576ed54ee" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:44.977289 9593 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s user 0.014s sys 0.017s
W20260503 13:58:44.977353 9602 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s user 0.028s sys 0.031s
W20260503 13:58:44.977463 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 820263us
W20260503 13:58:44.854735 9596 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.811s user 0.029s sys 0.019s
W20260503 13:58:44.980358 9596 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.941s user 0.037s sys 0.026s
W20260503 13:58:44.888770 9601 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.014s sys 0.036s
W20260503 13:58:44.980799 9601 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s user 0.018s sys 0.040s
I20260503 13:58:44.902127 10871 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:44.985144 11092 log.cc:927] Time spent T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
W20260503 13:58:44.877775 9598 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.004s sys 0.021s
W20260503 13:58:44.985553 9598 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s user 0.005s sys 0.030s
W20260503 13:58:44.882568 9594 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.015s sys 0.034s
W20260503 13:58:44.854913 9418 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.020s sys 0.026s
I20260503 13:58:44.986340 9450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 595) took 1265 ms. Trace:
W20260503 13:58:44.986515 9418 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.782s user 0.028s sys 0.032s
I20260503 13:58:44.987706 10879 raft_consensus.cc:493] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:44.988214 10879 raft_consensus.cc:515] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.980449 10193 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:44.989708 10193 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.991322 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.991657 10879 leader_election.cc:290] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:44.839007 10636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s user 0.014s sys 0.014s
W20260503 13:58:44.873596 11052 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.007s sys 0.024s
W20260503 13:58:44.891177 9520 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.008s sys 0.019s
W20260503 13:58:44.986289 9594 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s user 0.017s sys 0.040s
W20260503 13:58:44.993366 9520 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s user 0.014s sys 0.021s
I20260503 13:58:44.993757 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.994462 9418 leader_election.cc:304] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:44.992425 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.993453 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.999156 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:45.000079 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:44.916209 10544 leader_election.cc:290] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:45.004202 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:45.004477 9418 leader_election.cc:304] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:45.005244 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:45.006637 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:45.003588 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2804950dfaaa4c1aa2da4b8c47dbd97f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:44.850271 10920 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:44.884442 10998 leader_election.cc:290] T 0f951d26e4ed4c3bb307eb276954663b P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:44.901100 10654 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:44.852419 10732 consensus_queue.cc:1059] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260503 13:58:44.825819 11037 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.844221 10896 raft_consensus.cc:493] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:44.848106 10768 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:44.976044 9521 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.774s user 0.007s sys 0.024s
W20260503 13:58:45.030311 9521 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.829s user 0.007s sys 0.024s
W20260503 13:58:44.976343 9599 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.679s user 0.010s sys 0.015s
W20260503 13:58:44.854497 10916 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.041s sys 0.133s
I20260503 13:58:44.978009 9597 raft_consensus.cc:2468] T 030fcc1b19c3489ea0ca320576ed54ee P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 1.
I20260503 13:58:44.982792 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:45.032809 9523 raft_consensus.cc:2393] T 9b9eaf6f8db04058b132b139d110dd24 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 3: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:44.984985 9596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 555) took 2442 ms. Trace:
I20260503 13:58:44.986719 9450 rpcz_store.cc:276] 0503 13:58:43.720543 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.720701 (+ 158us) service_pool.cc:224] Handling call
0503 13:58:43.727460 (+ 6759us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:43.727811 (+ 351us) raft_consensus.cc:1534] Early marking committed up to index 3
0503 13:58:43.757603 (+ 29792us) log.cc:844] Serialized 14 byte log entry
0503 13:58:44.683376 (+925773us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:44.683836 (+ 460us) log.cc:844] Serialized 26 byte log entry
0503 13:58:44.683958 (+ 122us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:44.683989 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:44.684073 (+ 84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:44.986137 (+302064us) raft_consensus.cc:1685] finished
0503 13:58:44.986160 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:44.986286 (+ 126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14913,"prepare.queue_time_us":29629,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":88773,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":4517,"thread_start_us":910476,"threads_started":1,"wal-append.queue_time_us":1131077}
I20260503 13:58:44.992868 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:45.035421 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
W20260503 13:58:44.891224 9416 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.687s user 0.009s sys 0.017s
W20260503 13:58:45.036088 9416 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.831s user 0.012s sys 0.023s
W20260503 13:58:45.036624 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 878238us
I20260503 13:58:45.037761 9416 leader_election.cc:304] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6; no voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868
I20260503 13:58:44.825083 11039 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:44.903551 10993 leader_election.cc:290] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:44.925482 10852 leader_election.cc:290] T 96fdbf26d7134a238a95b243fe5b8aac P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 4 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:45.011395 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
W20260503 13:58:44.993301 11052 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.789s user 0.007s sys 0.027s
I20260503 13:58:44.931926 10445 leader_election.cc:290] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:45.014962 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:45.014650 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:45.013393 11104 raft_consensus.cc:493] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
W20260503 13:58:45.031258 9599 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s user 0.010s sys 0.015s
I20260503 13:58:45.023620 11059 consensus_queue.cc:1048] T 09c4dc2df168460e83f4f02b17667151 P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260503 13:58:45.033129 9596 rpcz_store.cc:276] 0503 13:58:42.542343 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.219394 (+677051us) service_pool.cc:224] Handling call
0503 13:58:43.219660 (+ 266us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:43.227418 (+ 7758us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:43.227448 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:43.232978 (+ 5530us) log.cc:844] Serialized 52 byte log entry
0503 13:58:44.008478 (+775500us) raft_consensus.cc:1645] Marking committed up to 4
0503 13:58:44.982796 (+974318us) log.cc:844] Serialized 14 byte log entry
0503 13:58:44.983976 (+ 1180us) log.cc:844] Serialized 14 byte log entry
0503 13:58:44.984680 (+ 704us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:44.984768 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:44.984811 (+ 43us) raft_consensus.cc:1685] finished
0503 13:58:44.984826 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:44.984938 (+ 112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":263,"prepare.queue_time_us":1010104,"raft.queue_time_us":16161,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":6835,"thread_start_us":1641579,"threads_started":2,"wal-append.queue_time_us":1118199}
I20260503 13:58:45.025704 9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:45.050573 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefaab375804454f943c82f15b8fd68b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:45.013247 10802 raft_consensus.cc:2804] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260503 13:58:45.232832 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:45.424141 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 386419us
I20260503 13:58:45.433688 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:45.050161 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1dfd5295058420a9da1c1080e5caef3" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:45.425642 9447 raft_consensus.cc:2468] T 8cc630de8e5547ceb5c178981ee99fc2 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
W20260503 13:58:45.434294 9525 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.012s sys 0.028s
I20260503 13:58:45.050226 9491 leader_election.cc:304] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
W20260503 13:58:45.434934 9525 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s user 0.012s sys 0.028s
I20260503 13:58:45.435127 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4a7792770d1446eafce219f5f28d7db" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:45.435350 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 385168us
I20260503 13:58:45.050565 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6b3066cc4e4411f91d1cde17a952170" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:45.013330 10960 raft_consensus.cc:2804] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260503 13:58:45.372431 9492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 498376us
W20260503 13:58:45.387621 9448 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.012s sys 0.015s
I20260503 13:58:45.436245 9446 raft_consensus.cc:2410] T b1dfd5295058420a9da1c1080e5caef3 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260503 13:58:45.436445 9448 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.679s user 0.012s sys 0.016s
I20260503 13:58:45.051775 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:45.437395 9449 raft_consensus.cc:2410] T 5b4c6d7380b242c2930533560310a3f4 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 3 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260503 13:58:45.438192 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d049f50b1b5745d3b2e3568ed86c16d8" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:45.028283 11094 log.cc:927] Time spent T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.131s user 0.000s sys 0.001s
I20260503 13:58:45.101727 9450 raft_consensus.cc:1275] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 4bea375a55034750a1f8804b0db97868: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:45.413712 10193 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:45.398059 10879 raft_consensus.cc:3055] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Stepping down as leader of term 4
W20260503 13:58:45.229025 11101 log.cc:927] Time spent T a17efa5a2f524bcb966ff62c4182503f P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.201s user 0.000s sys 0.000s
W20260503 13:58:45.392040 11100 log.cc:927] Time spent T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.343s user 0.004s sys 0.004s
I20260503 13:58:45.116149 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:45.032308 10916 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.828s user 0.041s sys 0.133s
I20260503 13:58:45.026113 10898 raft_consensus.cc:493] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:45.425256 9451 raft_consensus.cc:2468] T cefaab375804454f943c82f15b8fd68b P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 6.
I20260503 13:58:45.427260 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:45.483364 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:45.483952 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "754ac001a92e4dd6a0b6620803b83722" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:45.116640 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a17efa5a2f524bcb966ff62c4182503f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:45.489084 9444 raft_consensus.cc:2410] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260503 13:58:45.425673 9528 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 6.
I20260503 13:58:45.489274 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec9b187575714b4884a75ff6aa911e15" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:45.442768 9445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:45.435719 9447 raft_consensus.cc:2468] T c4a7792770d1446eafce219f5f28d7db P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 7.
I20260503 13:58:45.484745 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:45.448738 9449 raft_consensus.cc:2393] T d049f50b1b5745d3b2e3568ed86c16d8 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:45.288529 10998 raft_consensus.cc:493] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:45.451303 9418 leader_election.cc:304] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
W20260503 13:58:45.433881 10639 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.009s sys 0.010s
I20260503 13:58:45.050396 11065 raft_consensus.cc:2749] T 54bf4ee675fb4fcf800b317817906dcc P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:45.438414 9446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3046bdc111c5477eb4eabf09e7f4e755" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:45.201632 10851 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.023s sys 0.022s
W20260503 13:58:45.410432 11109 log.cc:927] Time spent T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.213s user 0.000s sys 0.007s
I20260503 13:58:45.481539 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:45.531174 9450 raft_consensus.cc:2410] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260503 13:58:45.029621 10896 raft_consensus.cc:3060] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:45.434252 9602 raft_consensus.cc:2468] T 0db0ab3576734ec18108a10f65e31341 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:45.532055 9602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 591) took 1150 ms. Trace:
I20260503 13:58:45.532371 9602 rpcz_store.cc:276] 0503 13:58:44.381864 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:45.433454 (+1051590us) service_pool.cc:224] Handling call
0503 13:58:45.531999 (+ 98545us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:45.536948 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:45.538852 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 571) took 1224 ms. Trace:
W20260503 13:58:45.487531 9600 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.015s sys 0.021s
W20260503 13:58:45.492640 9416 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 58546.3us
I20260503 13:58:45.441042 10960 raft_consensus.cc:493] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:45.050623 10885 consensus_queue.cc:1048] T d04573f959de406a9590a1603f633e7e P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.444s
W20260503 13:58:45.425352 9566 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:45.492955 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 56002.5us
I20260503 13:58:45.493841 9445 raft_consensus.cc:2410] T 7df5306f46674fdca645be800280031b P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260503 13:58:45.049415 11097 raft_consensus.cc:493] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:45.429877 10636 raft_consensus.cc:493] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:45.432971 10802 raft_consensus.cc:493] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260503 13:58:45.495733 9565 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260503 13:58:45.494885 10639 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s user 0.010s sys 0.010s
I20260503 13:58:45.480603 10650 raft_consensus.cc:2804] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:45.490545 9596 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:45.579780 9596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 580) took 1241 ms. Trace:
I20260503 13:58:45.580128 9596 rpcz_store.cc:276] 0503 13:58:44.337869 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:45.484484 (+1146615us) service_pool.cc:224] Handling call
0503 13:58:45.579714 (+ 95230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:45.050534 11075 consensus_queue.cc:1048] T 030fcc1b19c3489ea0ca320576ed54ee P 1cb63886040047559d3893923cc9aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.476s
I20260503 13:58:45.429370 11104 raft_consensus.cc:515] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:45.526629 10544 raft_consensus.cc:2804] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260503 13:58:45.549937 9600 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.659s user 0.016s sys 0.021s
I20260503 13:58:45.481791 10879 raft_consensus.cc:740] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4bea375a55034750a1f8804b0db97868, State: Running, Role: LEADER
W20260503 13:58:45.530964 10851 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.969s user 0.031s sys 0.023s
I20260503 13:58:45.026332 10920 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:45.494378 10898 raft_consensus.cc:515] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:45.528810 10998 raft_consensus.cc:515] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:45.482817 10649 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.009s sys 0.018s
I20260503 13:58:45.558048 9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:45.546720 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:45.581157 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:45.977560 9418 leader_election.cc:304] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:45.557211 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:45.625394 9521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 653) took 1427 ms. Trace:
W20260503 13:58:45.529610 9598 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.015s sys 0.030s
I20260503 13:58:45.625571 9452 raft_consensus.cc:2410] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260503 13:58:45.983146 9598 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.997s user 0.021s sys 0.041s
I20260503 13:58:45.704154 9595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 567) took 1411 ms. Trace:
W20260503 13:58:45.538883 10907 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.015s sys 0.035s
I20260503 13:58:45.960178 9448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 601) took 2235 ms. Trace:
I20260503 13:58:45.983052 9521 rpcz_store.cc:276] 0503 13:58:44.198286 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:44.198404 (+ 118us) service_pool.cc:224] Handling call
0503 13:58:44.200525 (+ 2121us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:44.200876 (+ 351us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:44.200904 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:44.201318 (+ 414us) log.cc:844] Serialized 26 byte log entry
0503 13:58:45.030910 (+829592us) raft_consensus.cc:1645] Marking committed up to 2
0503 13:58:45.030950 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:45.031013 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:45.260288 (+229275us) spinlock_profiling.cc:243] Waited 3.61 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:45.606288 (+346000us) raft_consensus.cc:1685] finished
0503 13:58:45.606312 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:45.625318 (+ 19006us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2221,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":8301824,"thread_start_us":829183,"threads_started":1,"wal-append.queue_time_us":847709}
W20260503 13:58:45.752421 9568 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:43752 dropped due to backpressure: service queue is full with 50 items [suppressed 6 similar messages]
I20260503 13:58:45.983202 9447 raft_consensus.cc:2468] T a5439e0e393b4112a26459eb56502fae P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 3.
W20260503 13:58:45.525279 11064 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.010s sys 0.017s
I20260503 13:58:45.996047 9448 rpcz_store.cc:276] 0503 13:58:43.724380 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:43.756490 (+ 32110us) service_pool.cc:224] Handling call
0503 13:58:43.778962 (+ 22472us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:43.785318 (+ 6356us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:43.785346 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:43.785830 (+ 484us) log.cc:844] Serialized 52 byte log entry
0503 13:58:44.750153 (+964323us) raft_consensus.cc:1645] Marking committed up to 5
0503 13:58:45.436817 (+686664us) log.cc:844] Serialized 14 byte log entry
0503 13:58:45.491139 (+ 54322us) log.cc:844] Serialized 14 byte log entry
0503 13:58:45.579941 (+ 88802us) spinlock_profiling.cc:243] Waited 4.63 ms on lock 0x7b1800363c40. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea01e1a 00007f9f480a35c9 00007f9f3e8c01f0 00007f9f3ea8ab24 00007f9f3ea8aa95 00007f9f41b23440 00007f9f41b35af6 00007f9f41b46667 00007f9f41b465e8
0503 13:58:45.959723 (+379782us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:45.959807 (+ 84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:45.959942 (+ 135us) raft_consensus.cc:1685] finished
0503 13:58:45.959960 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:45.960112 (+ 152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":144031,"prepare.queue_time_us":1126984,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":62,"raft.queue_time_us":169902,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":59281,"spinlock_wait_cycles":10658560,"thread_start_us":1985729,"threads_started":3,"wal-append.queue_time_us":1073139}
W20260503 13:58:45.735078 10925 consensus_meta.cc:287] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.977s user 0.004s sys 0.012s
W20260503 13:58:45.777722 10295 consensus_meta.cc:287] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.967s user 0.009s sys 0.000s
W20260503 13:58:45.642400 10880 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.011s sys 0.037s
I20260503 13:58:45.550274 10960 raft_consensus.cc:3060] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
W20260503 13:58:45.752822 11094 log.cc:927] Time spent T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
W20260503 13:58:45.633733 10891 consensus_meta.cc:287] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 1.238s user 0.012s sys 0.002s
W20260503 13:58:45.823203 11119 log.cc:927] Time spent T 0f951d26e4ed4c3bb307eb276954663b P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20260503 13:58:45.983292 10649 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.076s user 0.011s sys 0.024s
I20260503 13:58:45.582592 10650 raft_consensus.cc:493] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
W20260503 13:58:45.680753 10868 consensus_meta.cc:287] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a: Time spent flushing consensus metadata: real 1.418s user 0.013s sys 0.004s
W20260503 13:58:45.744095 11051 consensus_meta.cc:287] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868: Time spent flushing consensus metadata: real 0.889s user 0.009s sys 0.000s
W20260503 13:58:45.991080 11125 log.cc:927] Time spent T 23732c2c1b724f65b22c4332efbd7da2 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.358s user 0.000s sys 0.000s
W20260503 13:58:45.990132 11128 log.cc:927] Time spent T 3b48fff133f84ff7827806460a23803b P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.357s user 0.000s sys 0.001s
I20260503 13:58:45.981520 11102 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:45.582862 10733 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:45.583086 10802 raft_consensus.cc:3060] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 2 FOLLOWER]: Advancing to term 3
I20260503 13:58:45.984134 9595 rpcz_store.cc:276] 0503 13:58:44.293111 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:44.293245 (+ 134us) service_pool.cc:224] Handling call
0503 13:58:44.296618 (+ 3373us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:44.296922 (+ 304us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:44.296950 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:44.297349 (+ 399us) log.cc:844] Serialized 26 byte log entry
0503 13:58:44.977826 (+680477us) raft_consensus.cc:1645] Marking committed up to 4
0503 13:58:45.396277 (+418451us) log.cc:844] Serialized 14 byte log entry
0503 13:58:45.540015 (+143738us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:45.540094 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:45.703899 (+163805us) raft_consensus.cc:1685] finished
0503 13:58:45.703924 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:45.704085 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87080,"prepare.queue_time_us":447731,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":33187,"raft.queue_time_us":145026,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":14736,"thread_start_us":1092540,"threads_started":2,"wal-append.queue_time_us":719551}
W20260503 13:58:45.992102 11108 log.cc:927] Time spent T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.382s user 0.001s sys 0.000s
W20260503 13:58:45.995507 10907 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.064s user 0.018s sys 0.054s
I20260503 13:58:45.774106 11104 leader_election.cc:290] T 38f7f4127357496eada52ad754f934e1 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:45.982542 10544 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:45.965370 10732 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:46.009040 9602 raft_consensus.cc:2410] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260503 13:58:46.149303 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:46.149937 9450 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 4.
I20260503 13:58:46.009927 9598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 569) took 1715 ms. Trace:
I20260503 13:58:46.014262 9596 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 7.
I20260503 13:58:45.632683 10768 leader_election.cc:290] T d3dcac7c2d37456fb52797e62e01bdd9 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:45.638944 11039 leader_election.cc:290] T 23732c2c1b724f65b22c4332efbd7da2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
W20260503 13:58:46.002607 11064 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.113s user 0.010s sys 0.031s
I20260503 13:58:46.070994 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46714 (request call id 614) took 1036 ms. Trace:
I20260503 13:58:45.539407 9594 rpcz_store.cc:276] 0503 13:58:44.314390 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:44.314749 (+ 359us) service_pool.cc:224] Handling call
0503 13:58:44.346594 (+ 31845us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:44.349681 (+ 3087us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:44.349709 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:44.350162 (+ 453us) log.cc:844] Serialized 52 byte log entry
0503 13:58:44.993773 (+643611us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:45.395516 (+401743us) log.cc:844] Serialized 14 byte log entry
0503 13:58:45.538579 (+143063us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:45.538647 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:45.538682 (+ 35us) raft_consensus.cc:1685] finished
0503 13:58:45.538694 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:45.538803 (+ 109us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":144944,"raft.queue_time_us":158664,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":11317,"spinlock_wait_cycles":7410048,"thread_start_us":1020679,"threads_started":2,"wal-append.queue_time_us":857541}
I20260503 13:58:45.555846 10654 raft_consensus.cc:493] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:45.987263 11075 raft_consensus.cc:493] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:46.174033 11075 raft_consensus.cc:515] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.175920 11075 leader_election.cc:290] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:45.988993 11052 raft_consensus.cc:493] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:46.176607 11052 raft_consensus.cc:515] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } } peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } }
I20260503 13:58:46.178439 11052 leader_election.cc:290] T 32fd983d73354e649ff91e64f251e44f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:46.085530 9418 leader_election.cc:304] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:46.174417 10869 raft_consensus.cc:493] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:46.179270 10654 raft_consensus.cc:515] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:46.180061 9602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 582) took 1841 ms (client timeout 2380 ms). Trace:
I20260503 13:58:45.988674 10756 raft_consensus.cc:493] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:46.098464 9452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46714 (request call id 618) took 1062 ms. Trace:
I20260503 13:58:46.181113 10654 leader_election.cc:290] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:45.934594 10896 raft_consensus.cc:515] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.182087 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:46.182713 9594 raft_consensus.cc:2393] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:46.183084 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 599) took 1282 ms. Trace:
I20260503 13:58:46.183544 10896 leader_election.cc:290] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:46.008780 10880 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.999s user 0.011s sys 0.051s
I20260503 13:58:46.003046 11065 raft_consensus.cc:493] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:45.587206 10636 raft_consensus.cc:515] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:46.178601 11114 log.cc:927] Time spent T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.582s user 0.006s sys 0.000s
I20260503 13:58:46.007555 11059 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:46.180819 9452 rpcz_store.cc:276] 0503 13:58:45.035639 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:45.036839 (+ 1200us) service_pool.cc:224] Handling call
0503 13:58:46.098251 (+1061412us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b5400303754. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:46.098402 (+ 151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":374068992}
I20260503 13:58:46.069147 10445 raft_consensus.cc:2804] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260503 13:58:46.007692 11105 raft_consensus.cc:493] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4bea375a55034750a1f8804b0db97868)
I20260503 13:58:46.185007 11059 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.185345 11105 raft_consensus.cc:515] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.185874 10636 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:45.579797 11097 raft_consensus.cc:515] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.173681 9453 rpcz_store.cc:276] 0503 13:58:45.034545 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:45.036103 (+ 1558us) service_pool.cc:224] Handling call
0503 13:58:45.036371 (+ 268us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:45.036575 (+ 204us) raft_consensus.cc:1534] Early marking committed up to index 4
0503 13:58:45.565904 (+529329us) log.cc:844] Serialized 14 byte log entry
0503 13:58:46.070667 (+504763us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:46.070716 (+ 49us) raft_consensus.cc:1645] Marking committed up to 4
0503 13:58:46.070754 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:46.070816 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:46.070942 (+ 126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":93125,"prepare.queue_time_us":643238,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":150399,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":7405,"spinlock_wait_cycles":9934080,"thread_start_us":875449,"threads_started":2,"wal-append.queue_time_us":409280}
I20260503 13:58:46.186874 11059 leader_election.cc:290] T 26449f3ab09141ada50d9334532cb3ec P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:46.187544 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:46.083382 11069 raft_consensus.cc:493] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:46.188041 11065 raft_consensus.cc:515] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.188154 11069 raft_consensus.cc:515] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.098851 10650 raft_consensus.cc:3060] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:46.183350 9594 rpcz_store.cc:276] 0503 13:58:44.901086 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:46.181829 (+1280743us) service_pool.cc:224] Handling call
0503 13:58:46.183032 (+ 1203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:46.147143 9566 leader_election.cc:304] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:46.188541 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:46.148150 10544 raft_consensus.cc:3060] T b633f30c57174316b12d468e847ca831 P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:46.157297 11139 raft_consensus.cc:493] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:46.157944 9450 raft_consensus.cc:1275] T a74decf860264836999f054535dd35ab P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260503 13:58:46.156925 10732 raft_consensus.cc:515] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.150887 9596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 600) took 1247 ms. Trace:
W20260503 13:58:46.159658 11126 log.cc:927] Time spent T 5b4c6d7380b242c2930533560310a3f4 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.530s user 0.005s sys 0.002s
W20260503 13:58:46.161244 11123 log.cc:927] Time spent T 7df5306f46674fdca645be800280031b P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.536s user 0.004s sys 0.003s
I20260503 13:58:46.162231 9599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 566) took 1870 ms. Trace:
I20260503 13:58:46.144903 11051 raft_consensus.cc:493] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:46.132393 10879 consensus_queue.cc:260] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.166400 9527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 676) took 1369 ms. Trace:
I20260503 13:58:46.122292 10868 raft_consensus.cc:2749] T facca7cb510c4c7a8be9484f3622012f P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:46.130638 11140 raft_consensus.cc:493] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:46.131304 11102 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.097007 10891 raft_consensus.cc:515] T f3d3814bacf748a4928419aa5e860f4a P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.159137 10898 leader_election.cc:290] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:46.153882 10998 leader_election.cc:290] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:46.150663 9598 rpcz_store.cc:276] 0503 13:58:44.294259 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:44.294386 (+ 127us) service_pool.cc:224] Handling call
0503 13:58:44.294659 (+ 273us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:44.294982 (+ 323us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:44.295008 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:44.295389 (+ 381us) log.cc:844] Serialized 26 byte log entry
0503 13:58:44.985865 (+690476us) raft_consensus.cc:1645] Marking committed up to 4
0503 13:58:45.985231 (+999366us) log.cc:844] Serialized 14 byte log entry
0503 13:58:46.009605 (+ 24374us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:46.009677 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:46.009716 (+ 39us) raft_consensus.cc:1685] finished
0503 13:58:46.009728 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:46.009863 (+ 135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23066,"prepare.queue_time_us":543543,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":24419,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":55944,"thread_start_us":1713430,"threads_started":3,"wal-append.queue_time_us":691622,"wal-append.run_cpu_time_us":1278,"wal-append.run_wall_time_us":1005692}
I20260503 13:58:46.170714 10920 leader_election.cc:290] T 09c4dc2df168460e83f4f02b17667151 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:46.127211 10849 raft_consensus.cc:493] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:46.142992 11144 raft_consensus.cc:493] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:46.070179 10295 raft_consensus.cc:515] T d049f50b1b5745d3b2e3568ed86c16d8 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.144773 10733 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.157042 11138 raft_consensus.cc:493] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:46.183792 11039 raft_consensus.cc:2804] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260503 13:58:46.179026 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 93564.4us
I20260503 13:58:46.179059 10869 raft_consensus.cc:515] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.187700 11097 leader_election.cc:290] T facca7cb510c4c7a8be9484f3622012f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:46.191354 11105 leader_election.cc:290] T ec9b187575714b4884a75ff6aa911e15 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:46.128065 10193 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:45.997210 9568 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:43752 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 601)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 599)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 543)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 538)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 527)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 531)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 537)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 535)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 526)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 524)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 525)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 594)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 544)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 574)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 575)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 576)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 577)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 578)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 579)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 581)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 584)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 586)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 588)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 589)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 590)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 592)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 593)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 597)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 598)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 514)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 520)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 522)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 530)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 532)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 536)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 540)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43752 (request call id 542)
I20260503 13:58:46.169554 11137 raft_consensus.cc:493] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:46.297520 9522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c4dc2df168460e83f4f02b17667151" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:46.191865 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:46.310456 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 163396us
I20260503 13:58:46.294312 9528 raft_consensus.cc:1240] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:46.310032 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:46.310678 9452 raft_consensus.cc:2468] T 94b14392ab2e47fea655f24d4c3e5da5 P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 1.
I20260503 13:58:46.310704 9522 raft_consensus.cc:2410] T 09c4dc2df168460e83f4f02b17667151 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260503 13:58:46.190577 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:46.285226 9600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 573) took 1965 ms. Trace:
I20260503 13:58:46.297242 9521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a"
I20260503 13:58:46.314846 9449 raft_consensus.cc:2410] T b3b7ab9cca2847d392329f5c80956bbd P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260503 13:58:46.314787 9521 raft_consensus.cc:3060] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:46.313679 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:46.313908 9453 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 7.
I20260503 13:58:46.318410 9596 rpcz_store.cc:276] 0503 13:58:44.903032 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:45.580885 (+677853us) service_pool.cc:224] Handling call
0503 13:58:46.150842 (+569957us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:46.197490 9526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 614) took 1209 ms. Trace:
I20260503 13:58:46.318974 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:46.319296 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.319989 9596 raft_consensus.cc:2410] T 50d2bf9820c443d5b7d017429cb0a818 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260503 13:58:46.227138 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:46.313395 9527 rpcz_store.cc:276] 0503 13:58:44.796512 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:44.799043 (+ 2531us) service_pool.cc:224] Handling call
0503 13:58:44.799308 (+ 265us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:44.799598 (+ 290us) raft_consensus.cc:1534] Early marking committed up to index 1
0503 13:58:44.810462 (+ 10864us) log.cc:844] Serialized 14 byte log entry
0503 13:58:45.028864 (+218402us) spinlock_profiling.cc:243] Waited 4.38 ms on lock 0x7b1800414940. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea8da33 00007f9f3ea8fe01 00007f9f3ea8fdb9 00007f9f3ea8fd49 00007f9f3ea8fd11 00007f9f3ea8f00d 0000000000379254 0000000000379099 00007f9f3ea70fba
0503 13:58:45.357932 (+329068us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:45.426116 (+ 68184us) log.cc:844] Serialized 26 byte log entry
0503 13:58:45.426337 (+ 221us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:45.426397 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:45.426520 (+ 123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:46.166156 (+739636us) raft_consensus.cc:1685] finished
0503 13:58:46.166236 (+ 80us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:46.166349 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34082,"prepare.queue_time_us":10761,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":2021,"raft.queue_time_us":35627,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":46504,"spinlock_wait_cycles":10084224,"thread_start_us":446252,"threads_started":1,"wal-append.queue_time_us":821771}
I20260503 13:58:46.312985 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22875ad0517447a2a9afc25bf367486f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.323833 9598 raft_consensus.cc:2468] T 22875ad0517447a2a9afc25bf367486f P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 7.
I20260503 13:58:46.324389 9566 leader_election.cc:304] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:46.324718 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c480795fed14481acd9ca2d189b9583" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.313110 9600 rpcz_store.cc:276] 0503 13:58:44.319468 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:44.319584 (+ 116us) service_pool.cc:224] Handling call
0503 13:58:44.349893 (+ 30309us) raft_consensus.cc:1471] Updating replica for 2 ops
0503 13:58:44.350207 (+ 314us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:44.350233 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0503 13:58:44.363899 (+ 13666us) log.cc:844] Serialized 52 byte log entry
0503 13:58:44.879421 (+515522us) raft_consensus.cc:1645] Marking committed up to 4
0503 13:58:45.980924 (+1101503us) log.cc:844] Serialized 14 byte log entry
0503 13:58:45.981259 (+ 335us) log.cc:844] Serialized 14 byte log entry
0503 13:58:45.981446 (+ 187us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:45.982206 (+ 760us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:46.178052 (+195846us) spinlock_profiling.cc:243] Waited 52 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:46.284985 (+106933us) raft_consensus.cc:1685] finished
0503 13:58:46.285012 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:46.285163 (+ 151us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":280301,"prepare.run_cpu_time_us":7138,"prepare.run_wall_time_us":104172,"raft.queue_time_us":1012,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":11137,"spinlock_wait_cycles":119559424,"thread_start_us":1577702,"threads_started":2,"wal-append.queue_time_us":1115427}
I20260503 13:58:46.326915 9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:46.319380 9526 rpcz_store.cc:276] 0503 13:58:44.987980 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:45.000720 (+ 12740us) service_pool.cc:224] Handling call
0503 13:58:45.000983 (+ 263us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:45.001153 (+ 170us) raft_consensus.cc:1534] Early marking committed up to index 5
0503 13:58:45.072310 (+ 71157us) log.cc:844] Serialized 14 byte log entry
0503 13:58:45.252660 (+180350us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b1800415240. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f3ea01e1a 00007f9f480a35c9 00007f9f3e8c01f0 00007f9f3ea8ab24 00007f9f3ea8d242 00007f9f416c45ab 00007f9f416ca0c3 00007f9f416ca5da 00007f9f41b35e0a
0503 13:58:46.195892 (+943232us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:46.197072 (+ 1180us) spinlock_profiling.cc:243] Waited 60.9 ms on lock 0x7b1800415240. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f478f4251 00007f9f47900ab1 00007f9f416d285c 00007f9f3ea70c0b 00007f9f47869bc9 00007f9f3ea8ccdb 00007f9f3ea8af7f 00007f9f3ea8aa95 00007f9f41b23440 00007f9f41b35af6 00007f9f41b46667
0503 13:58:46.197176 (+ 104us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:46.197227 (+ 51us) raft_consensus.cc:1645] Marking committed up to 5
0503 13:58:46.197268 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:46.197333 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:46.197437 (+ 104us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":517707,"prepare.queue_time_us":84108,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":450599,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":28572,"spinlock_wait_cycles":190089728,"thread_start_us":226334,"threads_started":2,"wal-append.queue_time_us":920511}
I20260503 13:58:46.193128 9594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.313205 9599 rpcz_store.cc:276] 0503 13:58:44.291938 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:44.292070 (+ 132us) service_pool.cc:224] Handling call
0503 13:58:44.292377 (+ 307us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:44.296261 (+ 3884us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:44.296288 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:44.297364 (+ 1076us) log.cc:844] Serialized 26 byte log entry
0503 13:58:45.426318 (+1128954us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:45.481045 (+ 54727us) log.cc:844] Serialized 14 byte log entry
0503 13:58:45.812762 (+331717us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:45.812991 (+ 229us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:45.991601 (+178610us) spinlock_profiling.cc:243] Waited 38.8 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:46.161962 (+170361us) raft_consensus.cc:1685] finished
0503 13:58:46.162055 (+ 93us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:46.162180 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66967,"prepare.queue_time_us":187171,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":9241,"raft.queue_time_us":75224,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":22880,"spinlock_wait_cycles":89146240,"thread_start_us":1396293,"threads_started":2,"wal-append.queue_time_us":1312220}
I20260503 13:58:46.323391 9448 raft_consensus.cc:2468] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 5.
I20260503 13:58:46.331720 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5439e0e393b4112a26459eb56502fae" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.332790 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a763e2ddf6e7463bb123ce8d870a1afb" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.333674 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ebc0b241994520929db0a8b7fb8546" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:46.232362 9597 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 8.
I20260503 13:58:46.334626 9449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:46.337275 9449 raft_consensus.cc:3060] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:46.334246 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:46.337985 9453 raft_consensus.cc:3060] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Advancing to term 8
I20260503 13:58:46.330257 9598 raft_consensus.cc:2468] T 1c480795fed14481acd9ca2d189b9583 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
W20260503 13:58:46.335903 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 784927us
I20260503 13:58:46.339738 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8650e2ab26ac47f8a5a6cd4dd6c30d50" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.340157 9598 raft_consensus.cc:2468] T 8650e2ab26ac47f8a5a6cd4dd6c30d50 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:46.333297 9599 raft_consensus.cc:2468] T a763e2ddf6e7463bb123ce8d870a1afb P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 6.
W20260503 13:58:46.195183 11127 log.cc:927] Time spent T 38f7f4127357496eada52ad754f934e1 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.326s user 0.000s sys 0.009s
I20260503 13:58:46.345242 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:46.357744 9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63e583b228f24d97b0e2b31046675c77" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:46.340750 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322e401806f7452c9714ce855e35ca9b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.340958 9519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "facca7cb510c4c7a8be9484f3622012f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:46.346915 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b7ab9cca2847d392329f5c80956bbd" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.363229 9596 raft_consensus.cc:2468] T b3b7ab9cca2847d392329f5c80956bbd P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 6.
I20260503 13:58:46.365530 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.368499 9596 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 6.
I20260503 13:58:46.264093 10885 raft_consensus.cc:493] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:46.348243 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030fcc1b19c3489ea0ca320576ed54ee" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:46.198849 11115 log.cc:927] Time spent T facca7cb510c4c7a8be9484f3622012f P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.590s user 0.004s sys 0.002s
W20260503 13:58:46.193380 11107 log.cc:927] Time spent T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868: Append to log took a long time: real 0.744s user 0.002s sys 0.006s
I20260503 13:58:46.350315 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c918e4ac84c4238a06220bc2817286f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:46.340728 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:46.385394 9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:46.351619 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.386188 9523 raft_consensus.cc:2410] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20260503 13:58:46.386269 9600 raft_consensus.cc:2393] T 9b9eaf6f8db04058b132b139d110dd24 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 3: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
W20260503 13:58:46.352348 11120 log.cc:927] Time spent T 973019c2128c4039aa1d442714f23ae7 P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.743s user 0.010s sys 0.000s
I20260503 13:58:46.394920 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b633f30c57174316b12d468e847ca831" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.395502 9599 raft_consensus.cc:2468] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:46.352643 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0db0ab3576734ec18108a10f65e31341" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:46.353205 10732 leader_election.cc:290] T b633f30c57174316b12d468e847ca831 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:46.354293 9520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37228 (request call id 627) took 1356 ms. Trace:
I20260503 13:58:46.314235 11140 raft_consensus.cc:515] T f3d3814bacf748a4928419aa5e860f4a P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.354714 9491 leader_election.cc:304] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
W20260503 13:58:46.180409 9602 rpcz_store.cc:269] 0503 13:58:44.338314 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:45.546459 (+1208145us) service_pool.cc:224] Handling call
0503 13:58:46.179894 (+633435us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x7b54000ac154. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:46.180012 (+ 118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":60663552}
I20260503 13:58:46.413182 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b9644d56a7f4556ae897658749f02e4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.317639 11051 raft_consensus.cc:515] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
W20260503 13:58:46.195447 11131 log.cc:927] Time spent T 2804950dfaaa4c1aa2da4b8c47dbd97f P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.216s user 0.001s sys 0.000s
W20260503 13:58:46.196365 11135 log.cc:927] Time spent T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.193s user 0.000s sys 0.001s
I20260503 13:58:46.193308 10445 raft_consensus.cc:493] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:46.312528 11037 leader_election.cc:290] T 63e583b228f24d97b0e2b31046675c77 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:46.314705 11039 raft_consensus.cc:493] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260503 13:58:46.311110 11065 leader_election.cc:290] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:46.371668 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "973019c2128c4039aa1d442714f23ae7" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.346812 10879 raft_consensus.cc:3060] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:46.376242 9521 raft_consensus.cc:2468] T 5c55d864e0b0416ebdc09ce6e0310a8f P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 2.
I20260503 13:58:46.263273 11116 raft_consensus.cc:493] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:46.425586 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50d2bf9820c443d5b7d017429cb0a818" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:46.335556 11069 leader_election.cc:290] T 3046bdc111c5477eb4eabf09e7f4e755 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:46.380653 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5c2e56fa22a496280f2678cd6657c3d" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.427024 9596 raft_consensus.cc:2468] T d5c2e56fa22a496280f2678cd6657c3d P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
I20260503 13:58:46.427845 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7df5306f46674fdca645be800280031b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.428882 9596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d03f17c8e3a48fcbbb01ac5504d81fe" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:46.429919 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 105592us
I20260503 13:58:46.430711 9566 leader_election.cc:304] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:46.431041 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b4c6d7380b242c2930533560310a3f4" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.431866 9566 leader_election.cc:304] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:46.432406 9567 leader_election.cc:304] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:46.346735 10920 raft_consensus.cc:2749] T 2804950dfaaa4c1aa2da4b8c47dbd97f P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260503 13:58:46.316321 11144 raft_consensus.cc:515] T 26449f3ab09141ada50d9334532cb3ec P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.434983 9566 leader_election.cc:304] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:46.392516 9453 raft_consensus.cc:2468] T 05ba056a4d0c406a881cc34cb66078bc P 1cb63886040047559d3893923cc9aaf6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 8.
I20260503 13:58:46.317435 10193 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.393931 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b9eaf6f8db04058b132b139d110dd24" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6"
I20260503 13:58:46.437860 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:46.438537 9448 raft_consensus.cc:2410] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bea375a55034750a1f8804b0db97868 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260503 13:58:46.396260 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddaccadb96de46d3a03ca8d2f3a8c6e0" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.438973 9599 raft_consensus.cc:2468] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 7.
I20260503 13:58:46.439538 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.440001 9599 raft_consensus.cc:2393] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 2: Already voted for candidate 4bea375a55034750a1f8804b0db97868 in this term.
I20260503 13:58:46.424175 11039 raft_consensus.cc:3060] T ddaccadb96de46d3a03ca8d2f3a8c6e0 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Advancing to term 8
I20260503 13:58:46.180786 10756 raft_consensus.cc:515] T d3dcac7c2d37456fb52797e62e01bdd9 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.397701 10885 raft_consensus.cc:515] T a17efa5a2f524bcb966ff62c4182503f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.405259 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b937be606a421bab59946d15a81744" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.406143 9593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43754 (request call id 568) took 2112 ms. Trace:
I20260503 13:58:46.406677 9447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "497cd68663ac4d34b95230215aa13bd2" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:46.401165 9520 rpcz_store.cc:276] 0503 13:58:44.997461 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:45.229798 (+232337us) service_pool.cc:224] Handling call
0503 13:58:45.230036 (+ 238us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:46.328409 (+1098373us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:46.328439 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:46.328483 (+ 44us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:46.328511 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:46.354075 (+ 25564us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540034a154. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2c1e1 00007f9f41b2ac1e 00007f9f46afdad9 00007f9f40067d34 00007f9f40067cc2 00007f9f40067bf1 00007f9f40067b6c 00007f9f40066e22 00007f9f3fe65bcc 00007f9f3fe65006
0503 13:58:46.354121 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:46.354247 (+ 126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16225,"raft.queue_time_us":10414,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":13156,"spinlock_wait_cycles":2501549312}
I20260503 13:58:46.443374 9600 raft_consensus.cc:2468] T f5b937be606a421bab59946d15a81744 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 7.
I20260503 13:58:46.408445 9525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37220 (request call id 680) took 1544 ms. Trace:
W20260503 13:58:46.349779 10898 consensus_queue.cc:1175] T 6d03f17c8e3a48fcbbb01ac5504d81fe P 4bea375a55034750a1f8804b0db97868 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1cb63886040047559d3893923cc9aaf6" responder_term: 4 status { last_received { term: 4 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 4 index: 3 } }
I20260503 13:58:46.317328 10849 raft_consensus.cc:515] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.413772 9602 raft_consensus.cc:2468] T 8b9644d56a7f4556ae897658749f02e4 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 4.
W20260503 13:58:46.194376 11141 log.cc:927] Time spent T 83ebc0b241994520929db0a8b7fb8546 P 1cb63886040047559d3893923cc9aaf6: Append to log took a long time: real 0.219s user 0.001s sys 0.000s
I20260503 13:58:46.373410 10654 raft_consensus.cc:2804] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260503 13:58:46.316649 11137 raft_consensus.cc:515] T 3b48fff133f84ff7827806460a23803b P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.431957 11146 raft_consensus.cc:2804] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260503 13:58:46.422921 9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
W20260503 13:58:46.423825 9491 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.195:39997, user_credentials={real_user=slave}} blocked reactor thread for 69164.6us
I20260503 13:58:46.378240 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:46.452750 9595 raft_consensus.cc:3060] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Advancing to term 4
I20260503 13:58:46.435706 9598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23732c2c1b724f65b22c4332efbd7da2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868"
I20260503 13:58:46.336591 11138 raft_consensus.cc:515] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.421617 10445 raft_consensus.cc:3060] T 5c55d864e0b0416ebdc09ce6e0310a8f P 1cb63886040047559d3893923cc9aaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260503 13:58:46.436302 9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8e0a262a0c48d5b561d53f59fb808a" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:46.399706 9449 raft_consensus.cc:2468] T 96fdbf26d7134a238a95b243fe5b8aac P 1cb63886040047559d3893923cc9aaf6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:46.403295 9418 leader_election.cc:304] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a; no voters:
I20260503 13:58:46.335476 11139 raft_consensus.cc:515] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.436551 11157 raft_consensus.cc:2749] T 5b4c6d7380b242c2930533560310a3f4 P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260503 13:58:46.442266 9444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3dcac7c2d37456fb52797e62e01bdd9" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:46.442590 9448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb63886040047559d3893923cc9aaf6" is_pre_election: true
I20260503 13:58:46.402065 10802 raft_consensus.cc:515] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.444242 9447 raft_consensus.cc:2468] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 6.
I20260503 13:58:46.443733 9593 rpcz_store.cc:276] 0503 13:58:44.293721 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:44.293832 (+ 111us) service_pool.cc:224] Handling call
0503 13:58:44.295124 (+ 1292us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:44.295454 (+ 330us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:44.295480 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:44.295845 (+ 365us) log.cc:844] Serialized 26 byte log entry
0503 13:58:44.978244 (+682399us) raft_consensus.cc:1645] Marking committed up to 0
0503 13:58:44.978282 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:44.978350 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:46.192915 (+1214565us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:46.405892 (+212977us) raft_consensus.cc:1685] finished
0503 13:58:46.405913 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:46.406086 (+ 173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1473,"raft.run_cpu_time_us":52,"raft.run_wall_time_us":52,"spinlock_wait_cycles":424070912,"thread_start_us":682047,"threads_started":1,"wal-append.queue_time_us":969254}
I20260503 13:58:46.460268 9444 raft_consensus.cc:2468] T d3dcac7c2d37456fb52797e62e01bdd9 P 1cb63886040047559d3893923cc9aaf6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 7.
I20260503 13:58:46.446210 9525 rpcz_store.cc:276] 0503 13:58:44.864278 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:44.864388 (+ 110us) service_pool.cc:224] Handling call
0503 13:58:44.864691 (+ 303us) raft_consensus.cc:1471] Updating replica for 1 ops
0503 13:58:44.864872 (+ 181us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:44.864897 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0503 13:58:44.865306 (+ 409us) log.cc:844] Serialized 26 byte log entry
0503 13:58:45.435260 (+569954us) raft_consensus.cc:1645] Marking committed up to 3
0503 13:58:45.435297 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:45.435356 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0503 13:58:46.351896 (+916540us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x7b440002534c. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f3e8cb941 00007f9f3e99a375 00007f9f3e99ad92 00007f9f416d657b 00007f9f416d1e4f 00007f9f416c5c0e 00007f9f416c4f56 00007f9f416c4b78 00007f9f416ce5c1 00007f9f416ce579 00007f9f416ce509
0503 13:58:46.408185 (+ 56289us) raft_consensus.cc:1685] finished
0503 13:58:46.408207 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:46.408389 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":55944064,"thread_start_us":569673,"threads_started":1,"wal-append.queue_time_us":743813}
I20260503 13:58:46.461370 9448 raft_consensus.cc:3060] T 973019c2128c4039aa1d442714f23ae7 P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Advancing to term 6
W20260503 13:58:46.438273 11089 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.005s sys 0.045s
I20260503 13:58:46.292725 10960 raft_consensus.cc:515] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.440197 11037 raft_consensus.cc:2749] T b1dfd5295058420a9da1c1080e5caef3 P 4bea375a55034750a1f8804b0db97868 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260503 13:58:46.450670 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94b14392ab2e47fea655f24d4c3e5da5" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.572719 9600 raft_consensus.cc:2468] T 94b14392ab2e47fea655f24d4c3e5da5 P 4bea375a55034750a1f8804b0db97868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 1.
I20260503 13:58:46.450718 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c55d864e0b0416ebdc09ce6e0310a8f" candidate_uuid: "428b342dc4f041df956015e7d7cd173a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.441850 9599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cc630de8e5547ceb5c178981ee99fc2" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.456331 9453 raft_consensus.cc:1275] T a763e2ddf6e7463bb123ce8d870a1afb P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Refusing update from remote peer 428b342dc4f041df956015e7d7cd173a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260503 13:58:46.574092 9453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 638) took 1087 ms. Trace:
I20260503 13:58:46.574419 9453 rpcz_store.cc:276] 0503 13:58:45.486344 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:46.455895 (+969551us) service_pool.cc:224] Handling call
0503 13:58:46.456122 (+ 227us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:46.573814 (+117692us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:46.574036 (+ 222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:46.575587 9453 raft_consensus.cc:1240] T 2c918e4ac84c4238a06220bc2817286f P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 4. Current term is 5. Ops: []
I20260503 13:58:46.575911 10650 raft_consensus.cc:515] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.576295 10193 leader_election.cc:290] T 306069d785d24834b84d590925e644cf P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:46.576905 11149 raft_consensus.cc:493] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:46.577399 11149 raft_consensus.cc:515] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.577991 10650 leader_election.cc:290] T 030fcc1b19c3489ea0ca320576ed54ee P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:46.579020 11149 leader_election.cc:290] T 4d8e0a262a0c48d5b561d53f59fb808a P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:46.581786 11138 leader_election.cc:290] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:46.449868 10896 raft_consensus.cc:2749] T c6b3066cc4e4411f91d1cde17a952170 P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260503 13:58:46.568517 9598 raft_consensus.cc:3060] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Advancing to term 6
I20260503 13:58:46.451397 11146 raft_consensus.cc:493] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:46.585449 9447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 642) took 1098 ms. Trace:
I20260503 13:58:46.588102 9447 rpcz_store.cc:276] 0503 13:58:45.487262 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:46.460702 (+973440us) service_pool.cc:224] Handling call
0503 13:58:46.585049 (+124347us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:46.585189 (+ 140us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:46.585211 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:46.585248 (+ 37us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:46.585275 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:46.585321 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:46.585404 (+ 83us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:46.446115 10925 raft_consensus.cc:2804] T 5c55d864e0b0416ebdc09ce6e0310a8f P 4bea375a55034750a1f8804b0db97868 [term 2 FOLLOWER]: Leader election won for term 2
W20260503 13:58:46.568892 9567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 110011us
I20260503 13:58:46.569967 9594 raft_consensus.cc:2468] T 11b8ecb173e84681b8fb389c89395838 P 4bea375a55034750a1f8804b0db97868 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 3.
I20260503 13:58:46.448909 10654 raft_consensus.cc:493] T 8cc630de8e5547ceb5c178981ee99fc2 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 428b342dc4f041df956015e7d7cd173a)
I20260503 13:58:46.571071 9595 raft_consensus.cc:2468] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428b342dc4f041df956015e7d7cd173a in term 4.
I20260503 13:58:46.571522 9593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f951d26e4ed4c3bb307eb276954663b" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:46.572099 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.26.193:45599, user_credentials={real_user=slave}} blocked reactor thread for 126243us
W20260503 13:58:46.456876 9418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.26.194:46805, user_credentials={real_user=slave}} blocked reactor thread for 53654.3us
I20260503 13:58:46.456521 10639 raft_consensus.cc:493] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:46.454442 11069 raft_consensus.cc:493] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260503 13:58:46.586665 9444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46726 (request call id 643) took 1094 ms. Trace:
I20260503 13:58:46.461342 9524 raft_consensus.cc:2468] T 63e583b228f24d97b0e2b31046675c77 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:46.590777 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306069d785d24834b84d590925e644cf" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:46.429870 11116 raft_consensus.cc:515] T 306069d785d24834b84d590925e644cf P 1cb63886040047559d3893923cc9aaf6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
@ 0x7f9f3b31ec87 __libc_start_main at ??:0
@ 0x29847a _start at ??:?
I20260503 13:58:46.450471 10920 consensus_queue.cc:1048] T 32fd983d73354e649ff91e64f251e44f P 4bea375a55034750a1f8804b0db97868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.136s
I20260503 13:58:46.591089 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11b8ecb173e84681b8fb389c89395838" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:46.657266 10639 raft_consensus.cc:515] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.645200 11051 leader_election.cc:290] T b07d47a095894b958145d029c84c9123 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
I20260503 13:58:46.659286 11069 raft_consensus.cc:515] T d049f50b1b5745d3b2e3568ed86c16d8 P 428b342dc4f041df956015e7d7cd173a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb63886040047559d3893923cc9aaf6" member_type: VOTER last_known_addr { host: "127.8.26.193" port: 45599 } } peers { permanent_uuid: "428b342dc4f041df956015e7d7cd173a" member_type: VOTER last_known_addr { host: "127.8.26.194" port: 46805 } } peers { permanent_uuid: "4bea375a55034750a1f8804b0db97868" member_type: VOTER last_known_addr { host: "127.8.26.195" port: 39997 } }
I20260503 13:58:46.659659 10802 leader_election.cc:290] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [CANDIDATE]: Term 3 election: Requested vote from peers 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)
I20260503 13:58:46.639163 11149 raft_consensus.cc:493] T b633f30c57174316b12d468e847ca831 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260503 13:58:46.660112 9595 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 601) took 1735 ms (client timeout 2111 ms). Trace:
I20260503 13:58:46.659564 9444 rpcz_store.cc:276] 0503 13:58:45.491951 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:46.460837 (+968886us) service_pool.cc:224] Handling call
0503 13:58:46.586342 (+125505us) raft_consensus.cc:1471] Updating replica for 0 ops
0503 13:58:46.586446 (+ 104us) raft_consensus.cc:1534] Early marking committed up to index 0
0503 13:58:46.586459 (+ 13us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0503 13:58:46.586483 (+ 24us) raft_consensus.cc:1645] Marking committed up to 1
0503 13:58:46.586508 (+ 25us) raft_consensus.cc:1697] Filling consensus response to leader.
0503 13:58:46.586553 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
0503 13:58:46.586629 (+ 76us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260503 13:58:46.660915 9566 leader_election.cc:304] T cefaab375804454f943c82f15b8fd68b P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:46.657635 10733 leader_election.cc:290] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 428b342dc4f041df956015e7d7cd173a (127.8.26.194:46805)
W20260503 13:58:46.660482 9595 rpcz_store.cc:269] 0503 13:58:44.924883 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:46.346431 (+1421548us) service_pool.cc:224] Handling call
0503 13:58:46.660040 (+313609us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260503 13:58:46.571306 11089 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s user 0.005s sys 0.045s
I20260503 13:58:46.639312 9600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f7f4127357496eada52ad754f934e1" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
W20260503 13:58:46.664392 9491 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.26.195:39997 timed out after 2.108s (SENT) [suppressed 3 similar messages]
I20260503 13:58:46.665357 9600 raft_consensus.cc:2468] T 38f7f4127357496eada52ad754f934e1 P 4bea375a55034750a1f8804b0db97868 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 5.
I20260503 13:58:46.463193 11065 raft_consensus.cc:2804] T 497cd68663ac4d34b95230215aa13bd2 P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260503 13:58:46.594554 9451 raft_consensus.cc:2393] T 9b9eaf6f8db04058b132b139d110dd24 P 1cb63886040047559d3893923cc9aaf6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 428b342dc4f041df956015e7d7cd173a in current term 3: Already voted for candidate 1cb63886040047559d3893923cc9aaf6 in this term.
I20260503 13:58:46.666289 9520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b1804f75444731b8b3b6af3c038cf6" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:46.669656 9520 raft_consensus.cc:2468] T 57b1804f75444731b8b3b6af3c038cf6 P 428b342dc4f041df956015e7d7cd173a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea375a55034750a1f8804b0db97868 in term 5.
I20260503 13:58:46.670226 9594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43752 (request call id 517) took 1115 ms. Trace:
I20260503 13:58:46.665913 9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b07d47a095894b958145d029c84c9123" candidate_uuid: "4bea375a55034750a1f8804b0db97868" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "428b342dc4f041df956015e7d7cd173a" is_pre_election: true
I20260503 13:58:46.671298 9598 raft_consensus.cc:2468] T 23732c2c1b724f65b22c4332efbd7da2 P 4bea375a55034750a1f8804b0db97868 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb63886040047559d3893923cc9aaf6 in term 6.
I20260503 13:58:46.647485 9446 raft_consensus.cc:1240] T 322e401806f7452c9714ce855e35ca9b P 1cb63886040047559d3893923cc9aaf6 [term 6 FOLLOWER]: Rejecting Update request from peer 4bea375a55034750a1f8804b0db97868 for earlier term 5. Current term is 6. Ops: []
W20260503 13:58:46.647071 9597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43754 (request call id 583) took 2308 ms (client timeout 2108 ms). Trace:
W20260503 13:58:46.671918 9597 rpcz_store.cc:269] 0503 13:58:44.338523 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:45.035186 (+696663us) service_pool.cc:224] Handling call
0503 13:58:46.646902 (+1611716us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54003f11d4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:46.647009 (+ 107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2498478336}
I20260503 13:58:46.668015 9566 leader_election.cc:304] T 754ac001a92e4dd6a0b6620803b83722 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea375a55034750a1f8804b0db97868; no voters: 1cb63886040047559d3893923cc9aaf6, 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:46.670564 9594 rpcz_store.cc:276] 0503 13:58:45.554500 (+ 0us) service_pool.cc:167] Inserting onto call queue
0503 13:58:46.192860 (+638360us) service_pool.cc:224] Handling call
0503 13:58:46.670066 (+477206us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b540031bfd4. stack: 00007f9f3ea5bc83 00007f9f3ea5bb80 00007f9f3e3a9574 00007f9f480846b4 00007f9f48084649 00007f9f41a94951 00007f9f41b2f6da 00007f9f46aff1d7 00007f9f4006c834 00007f9f4006c7c2 00007f9f4006c6f1 00007f9f4006c66c 00007f9f4006b922 00007f9f3fe65bcc 00007f9f3fe65006 00007f9f3fe649af
0503 13:58:46.670169 (+ 103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":549832064}
I20260503 13:58:46.673563 9566 leader_election.cc:304] T c4a7792770d1446eafce219f5f28d7db P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb63886040047559d3893923cc9aaf6, 4bea375a55034750a1f8804b0db97868; no voters: 428b342dc4f041df956015e7d7cd173a
I20260503 13:58:46.663339 9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ba056a4d0c406a881cc34cb66078bc" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.674444 9595 raft_consensus.cc:2393] T 05ba056a4d0c406a881cc34cb66078bc P 4bea375a55034750a1f8804b0db97868 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 8: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
W20260503 13:58:46.591274 11100 log.cc:927] Time spent T 32fd983d73354e649ff91e64f251e44f P 428b342dc4f041df956015e7d7cd173a: Append to log took a long time: real 0.121s user 0.000s sys 0.008s
I20260503 13:58:46.677042 9567 leader_election.cc:304] T 57b1804f75444731b8b3b6af3c038cf6 P 4bea375a55034750a1f8804b0db97868 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428b342dc4f041df956015e7d7cd173a, 4bea375a55034750a1f8804b0db97868; no voters:
I20260503 13:58:46.681803 9597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fdbf26d7134a238a95b243fe5b8aac" candidate_uuid: "1cb63886040047559d3893923cc9aaf6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4bea375a55034750a1f8804b0db97868" is_pre_election: true
I20260503 13:58:46.682529 9597 raft_consensus.cc:2393] T 96fdbf26d7134a238a95b243fe5b8aac P 4bea375a55034750a1f8804b0db97868 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb63886040047559d3893923cc9aaf6 in current term 4: Already voted for candidate 428b342dc4f041df956015e7d7cd173a in this term.
I20260503 13:58:46.605070 11146 raft_consensus.cc:3060] T 0db0ab3576734ec18108a10f65e31341 P 428b342dc4f041df956015e7d7cd173a [term 4 FOLLOWER]: Advancing to term 5
I20260503 13:58:46.572045 10868 raft_consensus.cc:493] T d04573f959de406a9590a1603f633e7e P 428b342dc4f041df956015e7d7cd173a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cb63886040047559d3893923cc9aaf6)
I20260503 13:58:46.661027 10869 leader_election.cc:290] T c6b3066cc4e4411f91d1cde17a952170 P 428b342dc4f041df956015e7d7cd173a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cb63886040047559d3893923cc9aaf6 (127.8.26.193:45599), 4bea375a55034750a1f8804b0db97868 (127.8.26.195:39997)